Imaging device and signal processing device

ABSTRACT

The present disclosure relates to an imaging device and a signal processing device capable of expanding an application range of the imaging device. An imaging device includes an imaging element that includes one or more pixel output units that receive incident light from a subject incident without an intervention of an imaging lens or a pinhole and output one detection signal indicating an output pixel value modulated by an incident angle of the incident light, and outputs a detection signal set including one or more detection signals, and a communication unit that transmits imaging data including the detection signal set and position attitude data indicating at least one of a position or an attitude to a communication device by wireless communication. The present disclosure is applicable to, for example, a monitoring system and the like.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase of International PatentApplication No. PCT/JP2018/038943 filed on Oct. 19, 2018, which claimspriority benefit of Japanese Patent Application No. JP 2017-202861 filedin the Japan Patent Office on Oct. 19, 2017. Each of theabove-referenced applications is hereby incorporated herein by referencein its entirety.

TECHNICAL FIELD

The present disclosure relates to an imaging device and a signalprocessing device, and especially relates to an imaging device and asignal processing device capable of expanding an application range ofthe imaging device.

BACKGROUND ART

Conventionally, suggested is an imaging device that images whilemodulating light from a subject by a lattice-shaped optical filter thatcovers a light-receiving surface of an imaging element or an opticalfilter including a diffraction grating without using an imaging lens,and restores an image formed as an image of the subject by predeterminedarithmetic processing (refer to, for example, Non-Patent Document 1 andPatent Documents 1 and 2).

CITATION LIST Non-Patent Document

-   Non-Patent Document 1: M. Salman Asif and four others, “Flatcam:    Replacing lenses with masks and computation”, “2015 IEEE    International Conference on Computer Vision Workshop (ICCVW)”, 2015,    pages 663-666

Patent Document

-   Patent Document 1: Japanese Unexamined Patent Publication No.    2016-510910-   Patent Document 2: International Publication No. 2016/123529

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

By the way, the imaging device that does not use the imaging lens asdisclosed in Non-Patent Document 1 and Patent Documents 1 and 2 may bemade compact due to absence of the imaging lens, and application rangeof which is expected to be expanded.

The present disclosure is achieved in view of such a situation, and anobject thereof is to enlarge the application range of the imagingdevice.

Solution to Problems

An imaging device according to a first aspect of the present disclosureincludes an imaging element that includes one or more pixel output unitsthat receive incident light from a subject incident without anintervention of an imaging lens or a pinhole and output one detectionsignal indicating an output pixel value modulated by an incident angleof the incident light, and outputs a detection signal set including oneor more detection signals, and a communication unit that transmitsimaging data including the detection signal set and position attitudedata indicating at least one of a position or an attitude to acommunication device by wireless communication.

A signal processing device according to a second aspect of the presentdisclosure includes a restoration unit that restores a restored image byusing a plurality of detection signal sets included in a plurality ofimaging data from a plurality of imaging devices each including animaging element that includes one or more pixel output units thatreceive incident light from a subject incident without an interventionof an imaging lens or a pinhole and output one detection signalindicating an output pixel value modulated by an incident angle of theincident light, and outputs a detection signal set including one or moreof the detection signals.

In the first aspect of the present disclosure, incident light from asubject incident without an intervention of an imaging lens or a pinholeis received, a detection signal set including one or more detectionsignals indicating an output pixel value modulated by an incident angleof the incident light is output, and imaging data including thedetection signal set and position attitude data indicating at least oneof a position or an attitude is transmitted to a communication device bywireless communication.

In the second aspect of the present disclosure, a restored image isrestored by using a plurality of detection signal sets included in aplurality of imaging data from a plurality of imaging devices eachincluding an imaging element that includes one or more pixel outputunits that receive incident light from a subject incident without anintervention of an imaging lens or a pinhole and output one detectionsignal indicating an output pixel value modulated by an incident angleof the incident light, and outputs a detection signal set including oneor more of the detection signals.

Effects of the Invention

According to the first aspect or the second aspect of the presentdisclosure, the application range of the imaging device may be expanded.

Note that, the effects are not necessarily limited to the effects hereindescribed and may include any of the effects described in the presentdisclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a view for illustrating a principle of imaging in an imagingdevice to which a technology of the present disclosure is applied.

FIG. 2 is a block diagram illustrating a basic configuration example ofthe imaging device to which the technology of the present disclosure isapplied.

FIG. 3 is a view illustrating a configuration example of a pixel arrayunit of the imaging element in FIG. 2.

FIG. 4 is a view illustrating a first configuration example of theimaging element in

FIG. 2.

FIG. 5 is a view illustrating a second configuration example of theimaging element in FIG. 2.

FIG. 6 is a view for illustrating a principle of incident angledirectivity generation.

FIG. 7 is a view for illustrating a change in incident angle directivityusing an on-chip lens.

FIG. 8 is a view illustrating an example of a type of a light-shieldingfilm.

FIG. 9 is a view for illustrating a design of the incident angledirectivity.

FIG. 10 is a view for illustrating a difference between the on-chip lensand an imaging lens.

FIG. 11 is a view for illustrating a difference between the on-chip lensand the imaging lens.

FIG. 12 is a view for illustrating a difference between the on-chip lensand the imaging lens.

FIG. 13 is a view for illustrating a relationship between a subjectdistance and a coefficient indicating the incident angle directivity.

FIG. 14 is a view for illustrating a relationship between a narrow-anglepixel and a wide-angle pixel.

FIG. 15 is a view for illustrating a relationship between thenarrow-angle pixel and the wide-angle pixel.

FIG. 16 is a view for illustrating a relationship between thenarrow-angle pixel and the wide-angle pixel.

FIG. 17 is a view for illustrating a difference in image quality betweenthe narrow-angle pixel and the wide-angle pixel.

FIG. 18 is a view for illustrating a difference in image quality betweenthe narrow-angle pixel and the wide-angle pixel.

FIG. 19 is a view for illustrating an example of combining pixels of aplurality of angles of view.

FIG. 20 is a flowchart illustrating imaging processing by the imagingdevice in

FIG. 2.

FIG. 21 is a view for illustrating a method of reducing a processingload.

FIG. 22 is a view for illustrating the method of reducing the processingload.

FIG. 23 is a view for illustrating the method of reducing the processingload.

FIG. 24 is a view for illustrating the method of reducing the processingload.

FIG. 25 is a view for illustrating the method of reducing the processingload.

FIG. 26 is a block diagram illustrating a configuration example of animaging system to which the technology of the present disclosure isapplied.

FIG. 27 is a view for illustrating an example of a method of using theimaging system in FIG. 26.

FIG. 28 is a block diagram illustrating a first embodiment of theimaging device in FIG. 26.

FIG. 29 is a schematic diagram illustrating a configuration example ofan appearance of the first embodiment of the imaging device in FIG. 26.

FIG. 30 is a view illustrating an example of a pattern of the pixelarray unit of the imaging element in FIG. 28.

FIG. 31 is a view illustrating an example of the pattern of the pixelarray unit of the imaging element in FIG. 28.

FIG. 32 is a view illustrating an example of the pattern of the pixelarray unit of the imaging element in FIG. 28.

FIG. 33 is a block diagram illustrating a configuration example of asignal processing device in FIG. 26.

FIG. 34 is a flowchart for illustrating a first embodiment of processingof a signal processing unit in FIG. 26.

FIG. 35 is a view illustrating a first example of a data structure of apacket of imaging data.

FIG. 36 is a flowchart for illustrating processing of the imaging devicein FIG. 28.

FIG. 37 is a block diagram illustrating a second embodiment of theimaging device in FIG. 26.

FIG. 38 is a block diagram illustrating a third embodiment of theimaging device in FIG. 26.

FIG. 39 is a view for illustrating a relationship between a position ofthe imaging device and an incident angle of a light beam from a pointlight source.

FIG. 40 is a graph illustrating an example of incident angledirectivity.

FIG. 41 is a block diagram illustrating a fourth embodiment of theimaging device in FIG. 26.

FIG. 42 is a view illustrating a direction in which inclination of theimaging device is detected.

FIG. 43 is a flowchart for illustrating a second embodiment ofprocessing of a signal processing unit in FIG. 26.

FIG. 44 is a view illustrating a second example of a data structure of apacket of imaging data.

FIG. 45 is a flowchart for illustrating processing of the imaging devicein FIG. 41.

FIG. 46 is a view for illustrating a relationship between pixelorientation and a light-receiving sensitivity characteristic.

FIG. 47 is a block diagram illustrating a fifth embodiment of theimaging device in FIG. 26.

FIG. 48 is a view illustrating a third example of a data structure of apacket of imaging data.

FIG. 49 is a block diagram illustrating a sixth embodiment of theimaging device in FIG. 26.

FIG. 50 is a flowchart for illustrating a third embodiment of processingof a signal processing unit in FIG. 26.

FIG. 51 is a view illustrating a fourth example of a data structure of apacket of imaging data.

FIG. 52 is a flowchart for illustrating processing of the imaging devicein FIG. 49.

FIG. 53 is a schematic diagram illustrating a configuration example ofan appearance of a seventh embodiment of the imaging device in FIG. 26.

FIG. 54 is a view for illustrating an example of a countermeasure in acase where electric power is insufficient.

FIG. 55 is a view illustrating a variation of the imaging element inFIG. 5.

FIG. 56 is a view for illustrating a variation of a pixel output unit.

FIG. 57 is a view illustrating a variation of the imaging element.

FIG. 58 is a view illustrating a variation of the imaging element.

FIG. 59 is a view illustrating a variation of the imaging element.

FIG. 60 is a view illustrating a variation of the imaging element.

FIGS. 61A and 61B are views illustrating a variation of the imagingelement.

FIG. 62 is a block diagram illustrating a first variation of an imagingsystem to which the technology of the present disclosure is applied.

FIG. 63 is a block diagram illustrating a second variation of an imagingsystem to which the technology of the present disclosure is applied.

MODE FOR CARRYING OUT THE INVENTION

A preferred embodiment of the present disclosure is hereinafterdescribed in detail with reference to the accompanying drawings. Notethat, in this specification and the drawings, the components havingsubstantially the same functional configuration are assigned with thesame reference sign and the description thereof is not repeatedappropriately.

Furthermore, the description is given in the following order.

1. Overview of Imaging Device of Present Disclosure

2. Basic Configuration Example of Imaging Device of Present Disclosure

3. First Embodiment

4. Second Embodiment: Example of Providing Power Supply Unit

5. Third Embodiment: Example of Providing Power Supply Unit for SolarPower Generation

6. Fourth Embodiment: Example of Detecting Position and Inclination ofImaging Device

7. Fifth Embodiment: Example of Detecting Orientation of Imaging Device

8. Sixth Embodiment: Example in Which Position and Attitude of ImagingDevice May Be Changed

9. Seventh Embodiment: Example of Providing Light-receiving Surfaces onBoth Surfaces of Imaging Device

10. Variation

11. Others

1. Overview of Imaging Device of Present Disclosure

First, an overview of an imaging device of the present disclosure isdescribed.

In the imaging device of the present disclosure, as illustrated in anupper left part of FIG. 1, an imaging element 51 in which each pixel hasdetection sensitivity with incident angle directivity is used. Here,each pixel having the detection sensitivity with the incident angledirectivity has a light-receiving sensitivity characteristic accordingto an incident angle of incident light on each pixel different for eachpixel. However, the light-receiving sensitivity characteristics of allthe pixels need not be fully different, and the light-receivingsensitivity characteristics of some pixels may be the same.

Here, for example, each of all subjects is a set of point light sourcesand light is emitted from each point light source in all directions. Forexample, a subject surface 31 of a subject in the upper left part ofFIG. 1 includes point light sources PA to PC, and the point lightsources PA to PC emit a plurality of light beams of light intensities ato c, respectively, around. Furthermore, the imaging element 51hereinafter includes pixels having different incident angledirectivities in positions Pa to Pc (hereinafter referred to as pixelsPa to Pc).

In this case, as illustrated in the upper left part of FIG. 1, the lightbeams of the same light intensity emitted from the same point lightsource are incident on respective pixels of the imaging element 51. Forexample, the light beam of the light intensity a emitted from the pointlight source PA is incident on each of the pixels Pa to Pc of theimaging element 51. In contrast, the light beams emitted from the samepoint light source are incident on the respective pixels at differentincident angles. For example, the light beams from the point lightsource PA are incident on the pixels Pa to Pc at the different incidentangles.

Here, since the incident angle directivities of the pixels Pa to Pc aredifferent from one another, the light beams of the same light intensityemitted from the same point light source are detected with differentsensitivities by the respective pixels. As a result, the light beams ofthe same light intensity are detected at different detection signallevels by the respective pixels. For example, the detection signallevels for the light beams of the light intensity a from the point lightsource PA have different values among the pixels Pa to Pc.

Then, the light-receiving sensitivity level of each pixel for the lightbeam from each point light source is obtained by multiplying the lightintensity of the light beam by a coefficient indicating thelight-receiving sensitivity to the incident angle of the light beam(that is, the incident angle directivity). For example, the detectionsignal level of the pixel Pa for the light beam from the point lightsource PA is obtained by multiplying the light intensity a of the lightbeam of the point light source PA by the coefficient indicating theincident angle directivity of the pixel Pa to the incident angle of thelight beam on the pixel Pa.

Accordingly, detection signal levels DA, DB, and DC of the pixels Pc,Pb, and Pa are expressed by following equations (1) to (3),respectively.DA=α1×a+β1×b+γ1×c  (1)DB=α2×a+β2×b+γ2×c  (2)DC=α3×a+β3×b+γ3×c  (3)

Here, a coefficient α1 is the coefficient indicating the incident angledirectivity of the pixel Pc to the incident angle of the light beam fromthe point light source PA on the pixel Pc, and is set according to theincident angle. Furthermore, α1×a represents the detection signal levelof the pixel Pc to the light beam from the point light source PA.

A coefficient β1 is the coefficient indicating the incident angledirectivity of the pixel Pc to the incident angle of the light beam fromthe point light source PB on the pixel Pc, and is set according to theincident angle. Furthermore, β1×b represents the detection signal levelof the pixel Pc for the light beam from the point light source PB.

A coefficient γ1 is the coefficient indicating the incident angledirectivity of the pixel Pc to the incident angle of the light beam fromthe point light source PC to the pixel Pc, and is set according to theincident angle. Furthermore, γ1×c represents the detection signal levelof the pixel Pc to the light beam from the point light source PC.

In this manner, the detection signal level DA of the pixel Pa isobtained by the sum of products of the light intensities a, b, and c ofthe light beams from the point light sources PA, PB, and PC,respectively, in the pixel Pc, and the coefficients α1, β1, and γ1indicating the incident angle directivities according to the incidentangles.

Similarly, the detection signal level DB of the pixel Pb is obtained bythe sum of products of the light intensities a, b, and c of the lightbeams from the point light sources PA, PB, and PC, respectively, in thepixel Pb, and the coefficients α2, β2, and γ2 indicating the incidentangle directivities according to the incident angles as expressed byequation (2). Furthermore, the detection signal level DC of the pixel Pcis obtained by the sum of products of the light intensities a, b, and cof the light beams from the point light sources PA, PB, and PC,respectively, in the pixel Pa, and the coefficients α2, β2, and γ2indicating the incident angle directivities according to the incidentangles as expressed by equation (3).

However, in the detection signal levels DA, DB, and DC of the pixels Pa,Pb, and Pc, the light intensities a, b, and c of the light beams emittedfrom the point light sources PA, PB, and PC are mixed as expressed byequations (1) to (3). Therefore, as illustrated in an upper right partof FIG. 1, the detection signal level in the imaging element 51 isdifferent from the light intensity of each point light source on thesubject surface 31. Therefore, an image obtained by the imaging element51 is different from that formed as the image of the subject surface 31.

In contrast, the light intensities a to c of the light beams of therespective point light sources PA to PC are obtained by creatingsimultaneous equations including equations (1) to (3) and solving thecreated simultaneous equations. Then, by arranging pixels having pixelvalues according to the obtained light intensities a to c in accordancewith arrangement (relative positions) of the point light sources PA toPC, a restored image formed as the image of the subject surface 31 isrestored as illustrated in the lower right part of FIG. 1.

Note that, hereinafter, a set of coefficients (for example, coefficientsα1, β1, and γ1) for each of the equations forming the simultaneousequations is referred to as a coefficient set. Furthermore, hereinafter,a group of a plurality of coefficient sets (for example, coefficient setα1, β1, and γ1, coefficient set α2, β2, and γ2, and coefficient set α3,β3, and γ3) corresponding to a plurality of equations included in thesimultaneous equations is referred to as a coefficient set group.

In this manner, the imaging device including the imaging element 51 inwhich each pixel has the incident angle directivity as an indispensableconfiguration may be realized without need of an imaging lens, apinhole, and an optical filter disclosed in Patent Document 1 andNon-Patent Document 1 (hereinafter, referred to as Patent Document andthe like). As a result, the imaging lens, the pinhole, and the opticalfilter disclosed in Patent Document and the like are not theindispensable configurations, so that it is possible to make the imagingdevice short in height, that is, make a thickness thin in the lightincident direction in the configuration to realize an imaging function.

Furthermore, since the indispensable configuration is only the imagingelement 51, a degree of freedom in design may be improved. For example,in a conventional imaging device using the imaging lens, it is necessaryto arrange the pixels of the imaging element into a two-dimensionalarray in accordance with a position in which the image of the subject isformed by the imaging lens; however, this is not necessary in theimaging device using the imaging element 51. Therefore, a degree offreedom in arrangement of each pixel is improved, and for example, eachpixel may be freely arranged within a range in which light from thesubject is incident. For example, it becomes possible to arrange therespective pixels in a circular region, in a hollow square(square-shaped) region, or distribute in a plurality of regions.

Then, regardless of the arrangement of the respective pixels, it ispossible to obtain the light intensity of the light beam from each pointlight source by creating the simultaneous equations expressed byequations (1) to (3) described above by using the coefficients accordingto the incident angles of the light beams from the respective pointlight sources on the subject surface 31 on the respective pixels andsolving the same. Then, by arranging the pixels having the pixel valuesaccording to the obtained light intensities of the respective pointlight sources in accordance with the arrangement of the respective pointlight sources on the subject surface 31, it is possible to restore therestored image formed as the image of the subject surface 31.

2. Basic Configuration Example of Imaging Device of Present Disclosure

Next, a basic configuration example of the imaging device of the presentdisclosure is described with reference to FIGS. 2 to 25.

<Configuration Example of Imaging Device 101>

FIG. 2 is a block diagram illustrating a configuration example of animaging device 101 that is a basic imaging device to which thetechnology of the present disclosure is applied.

The imaging device 101 includes an imaging element 121, a restorationunit 122, a control unit 123, an input unit 124, a detection unit 125,an association unit 126, a display unit 127, a storage unit 128, arecording/playback unit 129, a recording medium 130, and a communicationunit 131. Furthermore, the restoration unit 122, the control unit 123,the input unit 124, the detection unit 125, the association unit 126,the display unit 127, the storage unit 128, the recording/playback unit129, the recording medium 130, and the communication unit 131 form asignal processing control unit 111 that performs signal processing andcontrol and the like of the imaging device 101. Note that, the imagingdevice 101 does not include an imaging lens (free from imaging lens).

Furthermore, the imaging element 121, the restoration unit 122, thecontrol unit 123, the input unit 124, the detection unit 125, theassociation unit 126, the display unit 127, the storage unit 128, therecording/playback unit 129, and the communication unit 131 areconnected to one another via a bus B1, and perform transmission andreception of data and the like via the bus B1. Note that, hereinafter,in order to simplify the description, description of the bus B1 in acase where each unit of the imaging device 101 performs transmission andreception of the data and the like via the bus B1 is omitted. Forexample, it is described that the input unit 124 supplies data to thecontrol unit 123 in a case where the input unit 124 supplies the data tothe control unit 123 via the bus B1.

The imaging element 121 corresponds to the imaging element 51 describedwith reference to FIG. 1, the imaging element that includes pixelshaving incident angle directivities and outputs an image includingdetection signals indicating the detection signal levels according to anamount of incident light to the restoration unit 122 or the bus B1.

More specifically, the imaging element 121 may have a basic structuresimilar to that of a general imaging element such as, for example, acomplementary metal oxide semiconductor (CMOS) image sensor. However, inthe imaging element 121, a configuration of each pixel forming a pixelarray is different from that of the general one, and the configurationis with the incident angle directivity as described later with referenceto FIGS. 3 to 5, for example. Then, the imaging element 121 has thelight-receiving sensitivity different (changing) depending on theincident angle of the incident light for each pixel and has the incidentangle directivity to the incident angle of the incident light in a pixelunit.

Note that, the image output from the imaging element 121 is an imageincluding the detection signals in which the image of the subject is notformed as illustrated in the upper right part of FIG. 1 described above,so that the subject cannot be visually recognized. That is, a detectionimage including the detection signals output by the imaging element 121is an image that is a set of pixel signals but with which a user cannotrecognize the subject through visual contact (the subject cannot bevisually recognized).

Therefore, hereinafter, the image including the detection signals inwhich the image of the subject is not formed as illustrated in the upperright part of FIG. 1, that is, the image captured by the imaging element121 is referred to as the detection image.

Note that, the imaging element 121 is not necessarily configured as thepixel array, and may also be configured as a line sensor, for example.Furthermore, the incident angle directivities need not necessarily bedifferent in a pixel unit, and the pixels having the same incident angledirectivity may be included.

The restoration unit 122 obtains, for example, a coefficient set groupcorresponding to a subject distance corresponding to a distance from theimaging element 51 to the subject surface 31 (subject surfacecorresponding to the restored image) in FIG. 1 and corresponding to theabove-described coefficients α1 to α3, β1 to β3, and γ1 to γ3 from thestorage unit 128. Furthermore, the restoration unit 122 creates thesimultaneous equations as expressed by equations (1) to (3) describedabove by using the detection signal level of each pixel of the detectionimage output from the imaging element 121 and the obtained coefficientset group. Then, the restoration unit 122 obtains the pixel value ofeach pixel forming the image formed as the image of the subjectillustrated in the lower right part of FIG. 1 by solving the createdsimultaneous equations. Therefore, an image in which the user mayrecognize the subject through the visual contact (in which the subjectmay be visually recognized) is restored from the detection image.Hereinafter, the image restored from the detection image is referred toas the restored image. However, in a case where the imaging element 121is sensitive only to light outside a visible wavelength band such asultraviolet rays, the restored image is not the image in which thesubject may be identified as a normal image, but this is also referredto as the restored image.

Furthermore, in the following, the restored image that is an image in astate in which the image of the subject is formed, the image beforecolor separation such as demosaic processing or synchronizationprocessing is referred to as a RAW image, and the detection imagecaptured by the imaging element 121 is distinguished therefrom as notbeing the RAW image although this is the image according to a colorfilter array.

Note that, the number of pixels of the imaging element 121 and thenumber of pixels forming the restored image are not necessarily thesame.

Furthermore, the restoration unit 122 performs demosaic processing, γcorrection, white balance adjustment, conversion processing to apredetermined compression format and the like on the restored image asnecessary. Then, the restoration unit 122 outputs the restored image tothe bus B1.

The control unit 123 includes, for example, various processors andcontrols each unit of the imaging device 101.

The input unit 124 includes an input device (for example, a key, aswitch, a button, a dial, a touch panel, a remote controller and thelike) for operating the imaging device 101, inputting data used forprocessing and the like. The input unit 124 outputs an operation signal,the input data and the like to the bus B1.

The detection unit 125 includes various sensors and the like used fordetecting states of the imaging device 101, the subject and the like.For example, the detection unit 125 includes an acceleration sensor anda gyro sensor that detect an attitude and movement of the imaging device101, a position detection sensor that detects a position of the imagingdevice 101 (for example, a global navigation satellite system (GNSS)receiver and the like), a ranging sensor that detects the subjectdistance and the like. The detection unit 125 outputs a signalindicating a detection result to the bus B1.

The association unit 126 associates the detection image obtained by theimaging element 121 with metadata corresponding to the detection image.The metadata includes, for example, the coefficient set group forrestoring the restored image using a target detection image, the subjectdistance and the like.

Note that, a method of associating the detection image with the metadatais not especially limited as long as a correspondence relationshipbetween the detection image and the metadata may be specified. Forexample, by assigning the metadata to image data including the detectionimage, assigning the same ID to the detection image and the metadata, orrecording the detection image and the metadata on the same recordingmedium 130, the detection image and the metadata are associated witheach other.

The display unit 127 includes a display, for example, and displaysvarious types of information (for example, the restored image and thelike). Note that, the display unit 127 may include an audio output unitsuch as a speaker to output audio.

The storage unit 128 includes one or more storage devices such as a readonly memory (ROM), a random access memory (RAM), and a flash memory, andstores, for example, programs, data and the like used for processing ofthe imaging device 101. For example, the storage unit 128 stores thecoefficient set group corresponding to the above-described coefficientsα1 to α3, β1 to β3, and γ1 to γ3 in association with various subjectdistances. More specifically, for example, the storage unit 128 storesthe coefficient set group including the coefficient for each pixel 121 aof the imaging element 121 for each point light source set on thesubject surface 31 for each subject surface 31 at each subject distance.

The recording/playback unit 129 records data on the recording medium 130and plays back (reads out) the data recorded on the recording medium130. For example, the recording/playback unit 129 records the restoredimage on the recording medium 130 or reads out the same from therecording medium 130. Furthermore, for example, the recording/playbackunit 129 records the detection image and the corresponding metadata onthe recording medium 130 or reads out the same from the recording medium130.

The recording medium 130 includes, for example, any of a hard disk drive(HDD), a solid state drive (SSD), a magnetic disk, an optical disk, amagneto-optical disk, a semiconductor memory or the like, a combinationthereof or the like.

The communication unit 131 communicates with other devices (for example,other imaging device, signal processing device and the like) by apredetermined communication method. Note that, the communication methodof the communication unit 131 may be wired or wireless. Furthermore, thecommunication unit 131 may support a plurality of communication methods.

<First Configuration Example of Imaging Element 121>

Next, with reference to FIGS. 3 and 4, a first configuration example ofthe imaging element 121 of the imaging device 101 in FIG. 2 isdescribed.

FIG. 3 illustrates a front view of a part of a pixel array unit of theimaging element 121. Note that, FIG. 3 illustrates an example of a casewhere the number of pixels in the pixel array unit is six pixelsvertically×six pixels horizontally, but the number of pixels in thepixel array unit is not limited to this.

In the imaging element 121 in FIG. 3, each pixel 121 a includes alight-shielding film 121 b that is one of modulation elements so as tocover a part of a light-receiving region (light-receiving surface) of aphotodiode thereof, and the incident light incident on each pixel 121 ais optically modulated according to the incident angle. Then, forexample, by providing the light-shielding film 121 b in a differentrange for each pixel 121 a, the light-receiving sensitivity to theincident angle of the incident light differs for each pixel 121 a, andthe respective pixels 121 a have different incident angle directivities.

For example, a light-shielding range of the light-receiving region ofthe photodiode is different between pixels 121 a-1 and 121 a-2 due toprovided light-shielding films 121 b-1 and 121 b-2 (at least any one ofthe light-shielding region (position) or a light-shielding area isdifferent). That is, in the pixel 121 a-1, the light-shielding film 121b-1 is provided so as to shield a part of a left side of thelight-receiving region of the photodiode by a predetermined width. Incontrast, in the pixel 121 a-2, the light-shielding film 121 b-2 isprovided so as to shield a part of a right side of the light-receivingregion by a predetermined width. Note that, the width by which thelight-shielding film 121 b-1 shields the light-receiving region of thephotodiode may be different from/the same as the width by which thelight-shielding film 121 b-2 shields the light-receiving region of thephotodiode. In the other pixels 121 a as well, similarly, thelight-shielding film 121 b is randomly arranged in the pixel array so asto shield a different range of the light-receiving region for eachpixel.

Note that, as a ratio of the light-shielding film 121 b covering thelight-receiving region of each pixel increases, the amount of light thatthe photodiode may receive decreases. Therefore, an area of thelight-shielding film 121 b is desirably the area that may secure adesired light amount, and may be limited to a maximum of about ¾ of thelight-receiving region, for example. With this arrangement, it becomespossible to secure the light amount not smaller than the desired amount.However, if each pixel includes an unshielded range by a widthcorresponding to a wavelength of the light to be received, it ispossible to receive a minimum amount of light. That is, for example, ina case of a blue pixel (B pixel), the wavelength is about 500 nm, and itis possible to receive the minimum amount of light if the light is notshielded beyond a width corresponding to this wavelength.

An upper stage of FIG. 4 is a side cross-sectional view of the firstconfiguration example of the imaging element 121, and a middle stage ofFIG. 4 is a top view of the first configuration example of the imagingelement 121. Furthermore, the side-cross sectional view in the upperstage of FIG. 4 is an AB cross-section in the middle stage of FIG. 4.Moreover, a lower stage of FIG. 4 is a circuit configuration example ofthe imaging element 121.

In the imaging element 121 in the upper stage of FIG. 4, the incidentlight is incident from an upper side downward in the drawing. Theadjacent pixels 121 a-1 and 121 a-2 are so-called backsideirradiation-type with a wiring layer Z12 provided in a lowermost layerin the drawing and a photoelectric conversion layer Z11 providedthereon.

Note that, in a case where there is no need to distinguish between thepixels 121 a-1 and 121 a-2, description of a number at the end of thereference sign is omitted, and they are simply referred to as the pixels121 a. Hereinafter, in the specification, a number at the end of thereference sign is sometimes omitted similarly for other configurations.

Furthermore, FIG. 4 illustrates only the side view and the top view oftwo pixels forming the pixel array of the imaging element 121; it goeswithout saying that more pixels 121 a are arranged but not illustrated.

Moreover, the pixels 121 a-1 and 121 a-2 include photodiodes 121 e-1 and121 e-2 in the photoelectric conversion layer Z11, respectively.Furthermore, on the photodiodes 121 e-1 and 121 e-2, on-chip lenses 121c-1 and 121 c-2 and color filters 121 d-1 and 121 d-2 are stacked fromabove, respectively.

The on-chip lenses 121 c-1 and 121 c-2 condense the incident light onthe photodiodes 121 e-1 and 121 e-2, respectively.

The color filters 121 d-1 and 121 d-2 are optical filters that transmitlight of specific wavelengths such as red, green, blue, infrared, andwhite, for example. Note that, in a case of white, the color filters 121d-1 and 121 d-2 may be transparent filters or they are not required.

In the photoelectric conversion layer Z11 of the pixels 121 a-1 and 121a-2, light-shielding films 121 g-1 to 121 g-3 are formed at boundariesbetween the pixels, and incident light L is inhibited from beingincident on the adjacent pixel to generate crosstalk as illustrated inFIG. 4, for example.

Furthermore, as illustrated in the upper and middle stages of FIG. 4,the light-shielding films 121 b-1 and 121 b-2 shield a part of alight-receiving surface S as seen from above. In the light-receivingsurface S of the photodiodes 121 e-1 and 121 e-2 in the pixels 121 a-1and 121 a-2, different ranges are shielded by the light-shielding films121 b-1 and 121 b-2, so that the different incident angle directivity isset independently for each pixel. However, the light-shielding rangedoes not have to be different in all the pixels 121 a of the imagingelement 121, and there may also be the pixels 121 a in which the samerange is shielded.

Note that, as illustrated in the upper stage of FIG. 4, thelight-shielding film 121 b-1 and the light-shielding film 121 g-1 areconnected to each other and formed into an L shape as seen from theside. Similarly, the light-shielding film 121 b-2 and thelight-shielding film 121 g-2 are connected to each other and formed intoan L shape as seen from the side. Furthermore, the light-shielding films121 b-1, 121 b-2, and 121 g-1 to 121 g-3 are formed using metal, forexample, tungsten (W), aluminum (Al), or an alloy of Al and copper (Cu).Furthermore, the light-shielding films 121 b-1, 121 b-2, and 121 g-1 to121 g-3 may be simultaneously formed using the same metal as that ofwiring by the same process as the process by which the wiring is formedin a semiconductor process. Note that, the light-shielding films 121b-1, 121 b-2, and 121 g-1 to 121 g-3 do not necessarily have the samethickness depending on the position.

Furthermore, as illustrated in the lower stage of FIG. 4, the pixel 121a includes a photodiode 161 (corresponding to the photodiode 121 e), atransfer transistor 162, a floating diffusion (FD) unit 163, a selectiontransistor 164, an amplification transistor 165, and a reset transistor166, and is connected to a current source 168 via a vertical signal line167.

The photodiode 161 is such that an anode electrode is grounded and acathode electrode is connected to a gate electrode of the amplificationtransistor 165 via the transfer transistor 162.

The transfer transistor 162 is driven according to a transfer signal TG.For example, when the transfer signal TG supplied to a gate electrode ofthe transfer transistor 162 reaches a high level, the transfertransistor 162 is turned on. Therefore, charges accumulated in thephotodiode 161 are transferred to the FD unit 163 via the transfertransistor 162.

The amplification transistor 165 serves as an input unit of a sourcefollower that is a readout circuit that reads out a signal obtained byphotoelectric conversion in the photodiode 161, and outputs a pixelsignal at a level corresponding to the charges accumulated in the FDunit 163 to the vertical signal line 167. That is, the amplificationtransistor 165 forms the source follower with the current source 168connected to one end of the vertical signal line 167 with a drainterminal connected to a power supply VDD and a source terminal connectedto the vertical signal line 167 via the selection transistor 164.

The FD unit 163 is a floating diffusion region having a chargecapacitance C1 provided between the transfer transistor 162 and theamplification transistor 165, and temporarily accumulates the chargestransferred from the photodiode 161 via the transfer transistor 162. TheFD unit 163 serves as a charge detection unit that converts the chargesinto a voltage, and the charges accumulated in the FD unit 163 areconverted into the voltage in the amplification transistor 165.

The selection transistor 164 is driven according to a selection signalSEL, turned on when the selection signal SEL supplied to a gateelectrode reaches a high level, and connects the amplificationtransistor 165 and the vertical signal line 167.

The reset transistor 166 is driven according to a reset signal RST. Forexample, the reset transistor 166 is turned on when a reset signal RSTsupplied to a gate electrode reaches a high level, discharges thecharges accumulated in the FD unit 163 to the power supply VDD, andresets the FD unit 163.

For example, the pixel circuit illustrated in the lower stage of FIG. 4operates as follows.

That is, as a first operation, the reset transistor 166 and the transfertransistor 162 are turned on, the charges accumulated in the FD unit 163are discharged to the power supply VDD, and the FD unit 163 is reset.

As a second operation, the reset transistor 166 and the transfertransistor 162 are turned off, an exposure period is started, and thecharges according to the amount of incident light are accumulated by thephotodiode 161.

As a third operation, after the reset transistor 166 is turned on andthe FD unit 163 is reset, the reset transistor 166 is turned off. Bythis operation, the FD unit 163 is set to reference potential.

As a fourth operation, the potential of the FD unit 163 in the resetstate is output from the amplification transistor 165 as the referencepotential.

As a fifth operation, the transfer transistor 162 is turned on, and thecharges accumulated in the photodiode 161 are transferred to the FD unit163.

As a sixth operation, the potential of the FD unit 163 to which thecharges of the photodiode are transferred is output from theamplification transistor 165 as signal potential.

Then, a signal obtained by subtracting the reference potential from thesignal potential by correlated double sampling (CDS) is output as adetection signal (pixel signal) of the pixel 121 a. A value of thisdetection signal (output pixel value) is modulated according to theincident angle of the incident light from the subject, and has thedifferent characteristic (directivity) depending on the incident angle(has the incident angle directivity).

<Second Configuration Example of Imaging Element 121>

FIG. 5 is a view illustrating a second configuration example of theimaging element 121. An upper stage of FIG. 5 is a side cross-sectionalview of the pixel 121 a of the imaging element 121 being the secondconfiguration example, and a middle stage of FIG. 5 is a top view of theimaging element 121. Furthermore, the side cross-sectional view in theupper stage of FIG. 5 is an AB cross-section in the middle stage of FIG.5. Moreover, a lower stage of FIG. 5 is a circuit configuration exampleof the imaging element 121.

The imaging element 121 in FIG. 5 has a configuration different fromthat of the imaging element 121 in FIG. 4 in that four photodiodes 121f-1 to 121 f-4 are formed in one pixel 121 a, and the light-shieldingfilm 121 g is formed in a region that separates the photodiodes 121 f-1to 121 f-4. That is, in the imaging element 121 in FIG. 5, thelight-shielding film 121 g is formed into a “+” shape as seen fromabove. Note that, the common configuration is assigned with the samereference sign as that in FIG. 4 and the detailed description thereof isomitted.

In the imaging element 121 in FIG. 5, occurrence of electrical andoptical crosstalk among the photodiodes 121 f-1 to 121 f-4 is preventedbecause the photodiodes 121 f-1 to 121 f-4 are separated by thelight-shielding film 121 g. That is, the light-shielding film 121 g inFIG. 5 is for preventing the crosstalk as is the case with thelight-shielding film 121 g of the imaging element 121 in FIG. 4 and isnot for providing the incident angle directivity.

Furthermore, in the imaging element 121 in FIG. 5, one FD unit 163 isshared by the four photodiodes 121 f-1 to 121 f-4. The lower stage ofFIG. 5 illustrates the circuit configuration example in which one FDunit 163 is shared by the four photodiodes 121 f-1 to 121 f-4. Notethat, in the lower stage of FIG. 5, the description of the configurationthe same as that in the lower stage of FIG. 4 is not repeated.

The lower stage of FIG. 5 differs from the circuit configuration in thelower stage of FIG. 4 in that photodiodes 161-1 to 161-4 (correspondingto the photodiodes 121 f-1 to 121 f-4 in the upper stage of FIG. 5) andtransfer transistors 162-1 to 162-4 are provided in place of thephotodiode 161 (corresponding to the photodiode 121 e in the upper stageof FIG. 4) and the transfer transistor 162, respectively, to share theFD unit 163.

With such a configuration, the charges accumulated in the photodiodes121 f-1 to 121 f-4 are transferred to the common FD unit 163 having apredetermined capacitance provided at a connection between thephotodiodes 121 f-1 to 121 f-4 and the gate electrode of theamplification transistor 165. Then, a signal corresponding to a level ofthe charges held in the FD unit 163 is read out as a detection signal(pixel signal) (however, the CDS processing is performed as describedabove).

Therefore, the charges accumulated by the photodiodes 121 f-1 to 121 f-4are allowed to selectively contribute to an output of the pixel 121 a,that is, the detection signal in various combinations. That is, it isconfigured such that the charges may be read out independently for eachof the photodiodes 121 f-1 to 121 f-4, and it is possible to obtain thedifferent incident angle directivities by making the photodiodes 121 f-1to 121 f-4 that contribute to the output (degree of contribution to theoutput of the photodiodes 121 f-1 to 121 f-4) different from each other.

For example, by transferring the charges of the photodiodes 121 f-1 and121 f-3 to the FD unit 163 and adding the signals obtained by readingout them, the incident angle directivity in a lateral direction may beobtained. Similarly, by transferring the charges of the photodiodes 121f-1 and 121 f-2 to the FD unit 163 and adding the signals obtained byreading out them, the incident angle directivity in a vertical directionmay be obtained.

Furthermore, a signal obtained on the basis of the charges selectivelyread out independently from the four photodiodes 121 f-1 to 121 f-4 isthe detection signal corresponding to one pixel forming the detectionimage.

Note that, contribution of (the charges of) each photodiode 121 f to thedetection signal may be realized, for example, not only by whether ornot to transfer the charges (detection value) of each photodiode 121 fto the FD unit 163, but also by using an electronic shutter function toreset the charges accumulated in the photodiode 121 f before thetransfer to the FD unit 163 and the like. For example, if the charges ofthe photodiode 121 f are reset immediately before the transfer to the FDunit 163, the photodiode 121 f does not contribute to the detectionsignal at all. On the other hand, when there is a time between the resetof the charges of the photodiode 121 f and the transfer of the chargesto the FD unit 163, the photodiode 121 f partially contributes to thedetection signal.

As described above, in a case of the imaging element 121 in FIG. 5, bychanging the combination of the photodiodes used for the detectionsignal out of the four photodiodes 121 f-1 to 121 f-4, it is possible toallow each pixel to have the different incident angle directivity.Furthermore, the detection signal output from each pixel 121 a of theimaging element 121 in FIG. 5 has a value (output pixel value) modulatedaccording to the incident angle of the incident light from the subject,and has the characteristic (directivity) different depending on theincident angle (has the incident angle directivity).

Note that, hereinafter, a unit of outputting the detection signalcorresponding to one pixel of the detection image is referred to as apixel output unit. The pixel output unit includes at least one or morephotodiodes, and each pixel 121 a of the imaging element 121 generallycorresponds to one pixel output unit.

For example, in the imaging element 121 in FIG. 4, since one pixel 121 aincludes one photodiode 121 e, one pixel output unit includes onephotodiode 121 e. In other words, one photodiode 121 e forms one pixeloutput unit.

Then, by making a light-shielding state by the light-shielding film 121b of each pixel 121 a different, the incident angle directivity of eachpixel output unit may be made different. Then, in the imaging element121 in FIG. 4, the incident light on each pixel 121 a is opticallymodulated using the light-shielding film 121 b, and as a result, thedetection signal of one pixel of the detection image reflecting theincident angle directivity is obtained by the signal output from thephotodiode 121 e of each pixel 121 a. That is, the imaging element 121in FIG. 4 includes a plurality of pixel output units that receives theincident light from the subject incident without an intervention of animaging lens or a pinhole, each pixel output unit includes onephotodiode 121 e, and the characteristic (incident angle directivity) tothe incident angle of the incident light from the subject is set foreach pixel output unit.

On the other hand, in the imaging element 121 in FIG. 5, one pixel 121 aincludes four photodiodes 121 f-1 to 121 f-4, so that one pixel outputunit includes four photodiodes 121 e. In other words, the fourphotodiodes 121 f forms one pixel output unit. On the other hand, eachphotodiode 121 e alone does not form an individual pixel output unit.

Then, as described above, the incident angle directivity for each pixeloutput unit is different by making the photodiode 121 f that contributesto the detection signal among the four photodiodes 121 f-1 to 121 f-4different for each pixel 121 a. That is, in the imaging element 121 inFIG. 5, a range that does not contribute to the output (detectionsignal) out of the four photodiodes 121 f-1 to 121 f-4 serves as thelight-shielding region. Then, the detection signal of one pixel of thedetection image reflecting the incident angle directivity is obtained bya combination of signals output from the photodiodes 121 f-1 to 121 f-4.That is, the imaging element 121 in FIG. 5 includes a plurality of pixeloutput units that receives the incident light from the subject incidentwithout an intervention of the imaging lens or the pinhole, each pixeloutput unit includes a plurality of photodiodes (for example,photodiodes 121 f-1 to 121 f-4), and (a degree of) the photodiodecontributing to the output is made different, so that a characteristicin each pixel output unit (incident angle directivity) to the incidentangle of the incident light from the subject is different from eachother.

Note that, in the imaging element 121 in FIG. 5, the incident light isincident on all the photodiodes 121 f-1 to 121 f-4 without beingoptically modulated, so that the detection signal is not the signalobtained by optical modulation. Furthermore, hereinafter, the photodiode121 f that does not contribute to the detection signal is also referredto as the photodiode 121 f that does not contribute to the pixel outputunit or output.

Note that, FIG. 5 illustrates an example in which the light-receivingsurface of the pixel output unit (pixel 121 a) is divided into fourequal parts, and the photodiode 121 f having the light-receiving surfaceof the same size is arranged in each region, that is, the example inwhich the photodiode is equally divided into four; however, the dividingnumber and dividing position of the photodiode may be arbitrarily set.

For example, the photodiode is not necessarily equally divided, and thedividing position of the photodiode may be different for each pixeloutput unit. Therefore, for example, even if the photodiode 121 f in thesame position is allowed to contribute to the output among a pluralityof pixel output units, the incident angle directivity differs betweenthe pixel output units. Furthermore, for example, by making the dividingnumber different between pixel output units, it becomes possible to setthe incident angle directivity more freely. Moreover, for example, boththe dividing number and the dividing position may be made differentbetween the pixel output units.

Furthermore, both the imaging element 121 in FIG. 4 and the imagingelement 121 in FIG. 5 have a configuration in which each pixel outputunit may independently set the incident angle directivity. In contrast,in the imaging device disclosed in Non-Patent Document 1 and PatentDocuments 1 and 2 described above, each pixel output unit of the imagingelement does not have a configuration in which the incident angledirectivity may be set independently. Note that, in the imaging element121 in FIG. 4, the incident angle directivity of each pixel output unitis set by the light-shielding film 121 b at the time of manufacture. Onthe other hand, in the imaging element 121 in FIG. 5, the dividingnumber and dividing position of the photodiode of each pixel output unitare set at the time of manufacture, but the incident angle directivityof each pixel output unit (combination of photodiodes allowed tocontribute to output) may be set at the time of use (for example, at thetime of imaging). Note that, in both the imaging element 121 in FIG. 4and the imaging element 121 in FIG. 5, it is not always necessary forall the pixel output units to have a configuration with the incidentangle directivity.

Note that, as described above, each pixel of the imaging elementnormally corresponds to one pixel output unit; however, as describedlater, there is a case where a plurality of pixels forms one pixeloutput unit. In the following, it is described assuming that each pixelof the imaging element corresponds to one pixel output unit unlessotherwise specified.

<Principle of Causing Incident Angle Directivity>

The incident angle directivity of each pixel of the imaging element 121occurs, for example, by a principle illustrated in FIG. 6. Note that, aleft upper part and a right upper part of FIG. 6 are views forillustrating the principle of occurrence of the incident angledirectivity in the imaging element 121 in FIG. 4, and a lower left partand a lower right part of FIG. 6 are views for illustrating theprinciple of occurrence of the incident angle directivity in the imagingelement 121 in FIG. 5.

Each of the pixels in the upper left part and upper right part of FIG. 6includes one photodiode 121 e. In contrast, each of the pixels in thelower left part and the lower right part of FIG. 6 includes twophotodiodes 121 f. Note that, herein, an example in which one pixelincludes two photodiodes 121 f is illustrated, but this is forconvenience in explanation, and the number of photodiodes 121 f includedin one pixel may be other than this.

In the pixel in the upper left part of FIG. 6, a light-shielding film121 b-11 is formed so as to shield a right half of a light-receivingsurface of a photodiode 121 e-11. Furthermore, in the pixel in the upperright part of FIG. 6, a light-shielding film 121 b-12 is formed so as toshield a left half of a light-receiving surface of a photodiode 121e-12. Note that, a dashed-dotted line in the drawing is an auxiliaryline that passes through the center in a horizontal direction of thelight-receiving surface of the photodiode 121 e and is perpendicular tothe light-receiving surface.

For example, in the pixel in the upper left part of FIG. 6, the incidentlight from the upper right that forms an incident angle 61 with respectto the dashed-dotted line in the drawing is easily received by a rangeon a left half not shielded by the light-shielding film 121 b-11 of thephotodiode 121 e-11. In contrast, the incident light from the upper leftthat forms an incident angle 62 with respect to the dashed-dotted linein the drawing is less easily received by the range on the left half notshielded by the light-shielding film 121 b-11 of the photodiode 121e-11. Accordingly, the pixel in the upper left part of FIG. 6 has theincident angle directivity with high light-receiving sensitivity for theincident light from the upper right in the drawing and lowlight-receiving sensitivity for the incident light from the upper left.

On the other hand, for example, in the pixel in the upper right part ofFIG. 6, the incident light from the upper right forming the incidentangle 61 is less easily received by a range in a left half shielded bythe light-shielding film 121 b-12 of the photodiode 121 e-12. Incontrast, the incident light from the upper left that forms the incidentangle 62 is easily received by a range in a right half not shielded bythe light-shielding film 121 b-12 of the photodiode 121 e-12.Accordingly, the pixel in the upper right part of FIG. 6 has theincident angle directivity with low light-receiving sensitivity for theincident light from the upper right in the drawing and highlight-receiving sensitivity for the incident light from the upper left.

Furthermore, the pixel in the lower left part of FIG. 6 includesphotodiodes 121 f-11 and 121 f-12 on left and right sides in thedrawing, and has a configuration with the incident angle directivitywithout the light-shielding film 121 b provided by reading out thedetection signal of one of them.

That is, in the pixel in the lower left part of FIG. 6, by reading outonly the signal of the photodiode 121 f-11 provided on the left side inthe drawing, the incident angle directivity similar to that of the pixelin the upper left part of FIG. 6 may be obtained. That is, the incidentlight from the upper right that forms the incident angle 61 with respectto the dashed-dotted line in the drawing is incident on the photodiode121 f-11, and a signal corresponding to the amount of received light isread out from the photodiode 121 f-11, so that this contributes to thedetection signal output from the pixel. In contrast, the incident lightfrom the upper left that forms the incident angle 62 with respect to thedashed-dotted line in the drawing is incident on the photodiode 121f-12, but this is not read out from the photodiode 121 f-12, so thatthis does not contribute to the detection signal output from the pixel.

Similarly, in a case where two photodiodes 121 f-13 and 121 f-14 areprovided as in the pixel in the lower right part in FIG. 6, by readingout only the signal of the photodiode 121 f-14 provided on the rightside in the drawing, the incident angle directivity similar to that ofthe pixel in the upper right part of FIG. 6 may be obtained. That is,the incident light from the upper right that forms the incident angle 61is incident on the photodiode 121 f-13, but the signal is not read outfrom the photodiode 121 f-13, so that this does not contribute to thedetection signal output from the pixel. In contrast, the incident lightfrom the upper left forming the incident angle 62 is incident on thephotodiode 121 f-14, and a signal corresponding to the amount ofreceived light is read out from the photodiode 121 f-14, so that thiscontributes to the detection signal output from the pixel.

Note that, in the pixel in the upper part of FIG. 6, the example inwhich the light-shielding range and the range not shielded are separatedin the central position in the horizontal direction of the pixel (thelight-receiving surface of the photodiode 121 e) is illustrated, but theranges may be separated in a position other than this. Furthermore, inthe pixel in the lower part of FIG. 6, the example in which the twophotodiodes 121 f are separated in the central position in thehorizontal direction of the pixel is illustrated, but they may beseparated in a position other than this. In this manner, by changing thelight-shielding range or the position in which the photodiode 121 f isseparated, the different incident angle directivities may be generated.

<Regarding Incident Angle Directivity in Configuration Including On-chipLens>

Next, the incident angle directivity in a configuration including theon-chip lens 121 c is described with reference to FIG. 7.

A graph in an upper stage of FIG. 7 illustrates the incident angledirectivity of the pixels in middle and lowerstages of FIG. 7. Notethat, the incident angle θ is plotted along the abscissa, and thedetection signal level is plotted along the ordinate. Note that, theincident angle θ is 0 degree in a case where the direction of theincident light coincides with a dashed-dotted line on a left side in themiddle stage of FIG. 7, an incident angle θ21 side on the left side inthe middle stage of FIG. 7 is a positive direction, and an incidentangle θ22 side on a right side in the middle stage of FIG. 7 is anegative direction. Therefore, the incident angle of the incident lightincident on the on-chip lens 121 c from the upper right is larger thanthat of the incident light incident from the upper left. That is, theincident angle θ increases as a travel direction of the incident lightinclines to the left (increases in the positive direction) and decreasesas this inclines to the right (increases in the negative direction).

Furthermore, the pixel in the left part in the middle stage of FIG. 7 isobtained by adding an on-chip lens 121 c-11 that condenses the incidentlight and a color filter 121 d-11 that transmits light of apredetermined wavelength to the pixel in the left part in the upperstage of FIG. 6. That is, in this pixel, the on-chip lens 121 c-11, thecolor filter 121 d-11, the light-shielding film 121 b-11, and thephotodiode 121 e-11 are stacked in this order in the light incidentdirection from the upper part of the drawing.

Similarly, the pixel in the right part in the middle stage of FIG. 7,the pixel in a left part in the lower stage of FIG. 7, and the pixel ina right part in the lower stage of FIG. 7 are obtained by adding theon-chip lens 121 c-11 and the color filter 121 d-11, or an on-chip lens121 c-12 and a color filter 121 d-12 to the pixel in the right part inthe upper stage of FIG. 6, the pixel in the left part in the lower stageof FIG. 6, and the pixel in the right part in the lower stage of FIG. 6,respectively.

In the pixel in the left part in the middle stage of FIG. 7, thedetection signal level (light-receiving sensitivity) of the photodiode121 e-11 changes according to the incident angle θ of the incident lightas indicated by a solid waveform in the upper stage of FIG. 7. That is,the larger the incident angle θ being the angle formed by the incidentlight with respect to the dashed-dotted line in the drawing (the largerthe incident angle θ in the positive direction (the more this inclinesrightward in the drawing)), the light is condensed in a range in whichthe light-shielding film 121 b-11 is not provided, so that the detectionsignal level of the photodiode 121 e-11 increases. In contrast, thesmaller the incident angle θ of the incident light (the larger theincident angle θ in the negative direction (the more this inclinesleftward in the drawing)), the light is condensed in a range in whichthe light-shielding film 121 b-11 is provided, so that the detectionsignal level of the photodiode 121 e-11 decreases.

Furthermore, in the pixel in the right part in the middle stage of FIG.7, the detection signal level (light-receiving sensitivity) of thephotodiode 121 e-12 changes according to the incident angle θ of theincident light as indicated by a dotted waveform in the upper stage ofFIG. 7. That is, the larger the incident angle θ of the incident light(the larger the incident angle θ in the positive direction), the lightis condensed in the range in which the light-shielding film 121 b-12 isprovided, so that the detection signal level of the photodiode 121 e-12decreases. In contrast, the smaller the incident angle θ of the incidentlight (the larger the incident angle θ in the negative direction), thelight is incident on the range in which the light-shielding film 121b-12 is not provided, so that the detection signal level of thephotodiode 121 e-12 increases.

The solid and dotted waveforms indicated in the upper stage of FIG. 7may be changed according to the range of the light-shielding film 121 b.Therefore, it becomes possible to allow the respective pixels to havethe different incident angle directivities depending on the range of thelight-shielding film 121 b.

As described above, the incident angle directivity is the characteristicof the light-receiving sensitivity of each pixel according to theincident angle θ, and this may also be said to be a characteristic of alight-shielding value according to the incident angle θ in the pixel inthe middle stage of FIG. 7. That is, the light-shielding film 121 bshields the incident light in a specific direction at a high level, butcannot sufficiently shield the incident light in other directions. Thischange in shielding level generates the detection signal level differentaccording to the incident angle θ as illustrated in the upper stage ofFIG. 7. Therefore, when the direction in which the light-shielding atthe highest level may be performed in each pixel is defined as thelight-shielding direction of each pixel, having the different incidentangle directivities in the respective pixels means having the differentlight-shielding directions in the respective pixels.

Furthermore, in the pixel in the left part in the lower stage of FIG. 7,as is the case with the pixel in the left part in the lower stage ofFIG. 6, by using the signal of only the photodiode 121 f-11 in the leftpart of the drawing, the incident angle directivity similar to that ofthe pixel in the left part in the middle stage of FIG. 7 may beobtained. That is, when the incident angle θ of the incident lightincreases (when the incident angle θ increases in the positivedirection), the detection signal level increases because the light iscondensed in the range of the photodiode 121 f-11 from which the signalis read out. In contrast, the smaller the incident angle θ of theincident light (the larger the incident angle θ in the negativedirection), the light is condensed in the range of the photodiode 121f-12 from which the signal is read out, so that the detection signallevel decreases.

Furthermore, similarly, in the pixel in the right part in the lowerstage of FIG. 7, as is the case with the pixel in the right part in thelower stage of FIG. 6, by using the signal of only a photodiode 121 f-14in the right part of the drawing, the incident angle directivity similarto that of the pixel in the right part in the middle stage of FIG. 7 maybe obtained. That is, when the incident angle θ of the incident lightincreases (when the incident angle θ increases in the positivedirection), the detection signal level per pixel decreases because thelight is condensed in the range of a photodiode 121 f-13 that does notcontribute to the output (detection signal). In contrast, the smallerthe incident angle θ of the incident light (the larger the incidentangle θ in the negative direction), the light is condensed in the rangeof the photodiode 121 f-14 that contributes to the output (detectionsignal), so that the detection signal level per pixel decreases.

Note that, as in the pixel in the lower stage of FIG. 7, in a pixelprovided with the plurality of photodiodes in the pixel and capable ofchanging the photodiode that contributes to the output, in order toallow each photodiode to have directivity to the incident angle of theincident light and generate the incident angle directivity in a pixelunit, the on-chip lens 121 c is the indispensable configuration for eachpixel.

Note that, as for the incident angle directivity, it is desirable thatrandomness is higher in a pixel unit. For example, if adjacent pixelshave the same incident angle directivity, equations (1) to (3) describedabove or equations (4) to (6) to be described later might be the sameequations, and as a result, the number of equations might beinsufficient for an unknown number that is a solution of thesimultaneous equations, and the pixel values forming the restored imagemight not be obtained.

Note that, in the following description, an example of a case of usingthe pixel 121 a that realizes the incident angle directivity using thelight-shielding film 121 b as the pixel 121 a in FIG. 4 is mainlydescribed. However, except for a case where the light-shielding film 121b is indispensable, it is also possible to use the pixel 121 a thatbasically divides the photodiode to realize the incident angledirectivity.

<Configuration of Light-shielding Film>

In the description above, as illustrated in FIG. 3, the example in whichthe entire light-receiving surface is shielded in the vertical directionand the light-shielding width and position in the horizontal directionare changed is illustrated as the configuration of the light-shieldingfilm 121 b of each pixel 121 a of the imaging element 121; however, as amatter of course, it is also possible to allow each pixel 121 a to havethe incident angle directivity by shielding the entire light-receivingsurface in the horizontal direction and changing the width (height) andposition in the vertical direction.

Note that, in the following, as illustrated in the example in FIG. 3,the light-shielding film 121 b that shields the entire light-receivingsurface of the pixel 121 a in the vertical direction and shields thelight-receiving surface by a predetermined width in the horizontaldirection is referred to as a lateral band-type light-shielding film 121b. Furthermore, the light-shielding film 121 b that shields the entirelight-receiving surface of the pixel 121 a in the horizontal directionand shields the light-receiving surface by a predetermined height in thevertical direction is referred to as a longitudinal band-typelight-shielding film 121 b.

Furthermore, as illustrated in a left part of FIG. 8, it is alsopossible to combine the longitudinal band-type and lateral band-typelight-shielding films 121 b, for example, to provide an L-shapedlight-shielding film 121 b for each pixel in the Bayer array.

Note that, in FIG. 8, a black range represents the light-shielding film121 b, and this is similarly displayed in the subsequent drawings unlessotherwise specified. Furthermore, in the example in FIG. 8, for each ofpixels 121 a-21 and 121 a-24 being green (G) pixels, a pixel 121 a-22being a red (R) pixel, and a pixel 121 a-23 being a blue (B) pixelforming the Bayer array, L-shaped light-shielding films 121 b-21 to 121b-24 are provided.

In this case, each pixel 121 a has the incident angle directivity asillustrated in the right part of FIG. 8. That is, in the right part ofFIG. 8, distribution of the light reception sensitivities of therespective pixels 121 a is illustrated in which the incident angle θx inthe horizontal direction (x direction) of the incident light is plottedalong the abscissa, and the incident angle θy in the vertical direction(y direction) of the incident light is plotted along the ordinate. Then,the light-receiving sensitivity within a range C4 is higher than thatoutside the range C4, the light-receiving sensitivity within a range C3is higher than that outside the range C3, the light-receivingsensitivity within a range C2 is higher than that outside the range C2,and the light-receiving sensitivity within a range C1 is higher thanthat outside the range C1.

Accordingly, in each pixel 121 a, the detection signal level to theincident light in which the incident angle θx in the horizontaldirection (x direction) and the incident angle θy in the verticaldirection (y direction) are within the range C1 is the highest. Then,the detection signal level decreases in the order of the incident lightin which the incident angle θx and the incident angle θy are within therange C2, the range C3, the range C4, and the range other than the rangeC4. Note that, intensity distribution of the light-receivingsensitivities illustrated in the right part of FIG. 8 is determined bythe range shielded by the light-shielding film 121 b in each pixel 121 aregardless of the Bayer array.

Note that, in the following, as the L-shaped light-shielding films 121b-21 to 121 b-24 in FIG. 8, the light-shielding film 121 b having ashape obtained by connecting the longitudinal band-type light-shieldingfilm and the lateral band-type light-shielding film at their ends iscollectively referred to as the L-shaped light-shielding film 121 b.

<Method of Setting Incident Angle Directivity>

Next, an example of a method of setting the incident angle directivityis described with reference to FIG. 9.

For example, a case where the light-shielding range in the horizontaldirection of the light-shielding film 121 b is a range from a left endof the pixel 121 a to a position A, and the light-shielding range in thevertical direction is a range from an upper end of the pixel 121 a to aposition B as illustrated in an upper stage of FIG. 9 is considered.

In this case, a weight Wx of 0 to 1, the weight according to theincident angle θx (deg) from the central position in the horizontaldirection of each pixel that serves as an index of the incident angledirectivity in the horizontal direction is set. In further detail, in acase where it is assumed that the weight Wx is 0.5 at the incident angleθx=θa corresponding to the position A, the weight Wx is set such thatthe weight Wx is 1 at the incident angle θx<θa−α, (−(θx−θa)/2α+0.5) atθa−α≤incident angle θx≤θa+α, and 0 at the incident angle θx>θa+α.

Similarly, a weight Wy of 0 to 1, the weight according to the incidentangle θy (deg) from the central position in the vertical direction ofeach pixel that serves as an index of the incident angle directivity inthe vertical direction is set. In further detail, in a case where it isassumed that the weight Wy is 0.5 at the incident angle θy=θbcorresponding to the position B, the weight Wy is set such that theweight Wy is 0 at the incident angle θy<θb−α, ((θy−θb)/2α+0.5) atθb−α≤incident angle θy≤θb+α, and 1 at the incident angle θy>θb+α.

Note that, the weight Wx and the weight Wy change as illustrated in thegraph in FIG. 9 in a case where an ideal condition is satisfied.

Then, by using the weights Wx and Wy obtained in this manner, it ispossible to obtain a coefficient corresponding to the incident angledirectivity, that is, the light-receiving sensitivity characteristic ofeach pixel 121 a. For example, a value obtained by multiplying theweight Wx corresponding to the incident angle θx of the incident lightfrom a certain point light source of the subject surface 31 by theweight Wy corresponding to the incident angle θy is set as thecoefficient for the point light source.

Furthermore, at that time, an inclination (½α) indicating the change inweight in the range in which the weight Wx in the horizontal directionand the weight Wy in the vertical direction are around 0.5 may be set byusing the on-chip lens 121 c having different focal distances.

For example, in a case where the focal distance of the on-chip lens 121c focuses on a surface of the light-shielding film 121 b as indicated bya solid line in a lower stage of FIG. 9, the inclination (½α) of theweight Wx in the horizontal direction and the weight Wy in the verticaldirection becomes steep. That is, the weight Wx and the weight Wydrastically change to 0 or 1 in the vicinity of a boundary of theincident angle in the horizontal direction θx=θa and the incident anglein the vertical direction θy=θb where the values are near 0.5.

Furthermore, for example, in a case where the focal distance of theon-chip lens 121 c focuses on the surface of the photodiode 121 e asindicated by a dotted line in the lower stage of FIG. 9, the inclination(½α) of the weight Wx in the horizontal direction and the weight Wy inthe vertical direction becomes gradient. That is, the weight Wx and theweight Wy gradually change to 0 or 1 in the vicinity of the boundary ofthe incident angle in the horizontal direction θx=θα and the incidentangle in the vertical direction θy=θb where the values are near 0.5.

For example, the focal distance of the on-chip lens 121 c changesdepending on a curvature of the on-chip lens 121 c. Therefore, by usingthe on-chip lens 121 c having different curvatures to change the focaldistance of the on-chip lens 121 c, it is possible to obtain differentincident angle directivities, that is, different light-receivingsensitivity characteristics.

Therefore, the incident angle directivity of the pixel 121 a may beadjusted by a combination of the range in which the photodiode 121 e isshielded by the light-shielding film 121 b and the curvature of theon-chip lens 121 c. Note that, the curvature of the on-chip lens may bethe same for all the pixels 121 a of the imaging element 121 or may bedifferent for some of the pixels 121 a.

For example, as an index indicating the incident angle directivity ofeach pixel 121 a of the imaging element 121, on the basis of theposition of each pixel 121 a, the shape, position, and range of thelight-shielding film 121 b of each pixel 121 a, the curvature of theon-chip lens 121 c and the like, the characteristics of the weight Wxand the weight Wy as illustrated in the graph of FIG. 9 are set for eachpixel 121 a. Furthermore, the incident angle of the light beam from thepoint light source to the pixel 121 a is obtained on the basis of apositional relationship between a certain point light source on thesubject surface 31 at a predetermined subject distance and a certainpixel 121 a of the imaging element 121. Then, the coefficient of thepixel 121 a for the point light source is obtained on the basis of theobtained incident angle and the characteristics of the weight Wx and theweight Wy of the pixel 121 a.

Similarly, by obtaining the coefficient as described above for thecombinations of the respective point light sources on the subjectsurface 31 and the respective pixels 121 a of the imaging element 121,the coefficient set group of the imaging element 121 for the subjectsurface 31 such as the coefficient sets α1, β1, and γ1, coefficient setα2, β2, and γ2, and coefficient set α3, β3, and γ3 in equations (1) to(3) described above may be obtained.

Note that, as described later with reference to FIG. 13, when thesubject distance from the subject surface 31 to the light-receivingsurface of the imaging element 121 is different, the incident angle ofthe light beam on the imaging element 121 from each point light sourceof the subject surface 31 is different, so that a different coefficientset group is required for each subject distance.

Furthermore, even on the subject surface 31 at the same subjectdistance, if the number and arrangement of the point light sources to beset are different, the incident angles of the light beams on the imagingelement 121 from the respective point light sources are different.Therefore, there is a case where a plurality of coefficient set groupsis required for the subject surface 31 at the same subject distance.Furthermore, the incident angle directivity of each pixel 121 a needs tobe set such that independence of the simultaneous equations describedabove may be secured.

<Difference Between On-Chip Lens and Imaging Lens>

In the imaging device 101 of the present disclosure, the imaging element121 has a configuration in which an optical block including the imaginglens or the pinhole is not required, but as described above, the on-chiplens 121 c is provided. Here, the on-chip lens 121 c and the imaginglens have different physical actions.

For example, as illustrated in FIG. 10, light incident on an imaginglens 152 out of diffused light emitted from a point light source P101 iscondensed at a pixel position P111 on the imaging element 151. That is,the imaging lens 152 is designed to condense the diffused light incidentat different angles from the point light source P101 at the pixelposition P111 to form an image of the point light source P101. The pixelposition P111 is specified by a principal light beam L101 passingthrough the point light source P101 and the center of the imaging lens152. [0155]

Furthermore, for example, as illustrated in FIG. 11, the light incidenton the imaging lens 152 out of the diffused light emitted from a pointlight source P102 different from the point light source P101 iscondensed at a pixel position P112 different from the pixel positionP111 on the imaging element 151. That is, the imaging lens 152 isdesigned to condense the diffused light incident at different anglesfrom the point light source P102 at the pixel position P112 to form animage of the point light source P102. The pixel position P112 isspecified by a principal light beam L102 passing through the point lightsource P102 and the center of the imaging lens 152.

In this manner, the imaging lens 152 forms the images of the point lightsources P101 and P102 having the different principal light beams at thedifferent pixel positions P111 and P112 on the imaging element 151,respectively.

Moreover, as illustrated in FIG. 12, in a case where the point lightsource P101 is present at infinity, a part of the diffused light emittedfrom the point light source P101 is incident on the imaging lens 152 asparallel light parallel to the principal light beam L101. For example,the parallel light including light beams between the light beams L121and L122 parallel to the principal light beam L101 is incident on theimaging lens 152. Then, the parallel light incident on the imaging lens152 is condensed at the pixel position P111 on the imaging element 151.That is, the imaging lens 152 is designed to condense the parallel lightfrom the point light source P101 present at infinity at the pixelposition P111 to form the image of the point light source P101.

Therefore, the imaging lens 152 has a condensing function to allow thediffused light from the point light source having a principal light beamincident angle 61 to be incident on a pixel (pixel output unit) P1 andallow the diffused light from the point light source having a principallight beam incident angle 62 different from the principal light beamincident angle 61 to be incident on a pixel (pixel output unit) P2different from the pixel P1, for example. That is, the imaging lens 152has the condensing function of allowing the diffused light from thelight sources having the different principal light beam incident anglesto be incident on a plurality of adjacent pixels (pixel output units).However, for example, the light beams from the point light sources closeto each other or the point light sources that are present at infinityand are substantially close to each other might be incident on the samepixel (pixel output unit).

In contrast, for example, as described with reference to FIGS. 4 and 5,the light passing through the on-chip lens 121 c is incident only on thelight-receiving surface of the photodiode 121 e or the photodiode 121 fforming the corresponding pixel (pixel output unit). In other words, theon-chip lens 121 c is provided for each pixel (pixel output unit), andcondenses the incident light incident thereon on only the correspondingpixel (pixel output unit). That is, the on-chip lens 121 c does not havethe condensing function of allowing the light beams from the differentpoint light sources to be incident on the different pixels (pixel outputunits).

Note that, in a case where the pinhole is used, a relationship betweenthe position of each pixel (pixel output unit) and the incident angle oflight is uniquely determined. Therefore, in a case of a configurationusing the pinhole and the conventional imaging element, it is notpossible to freely set the incident angle directivity independently foreach pixel.

<Relationship Between Subject Surface and Imaging Element>

Next, a relationship of the distance between the subject surface and theimaging element 121 is described with reference to FIG. 13.

Note that, as illustrated in an upper left part of FIG. 13, in a casewhere a subject distance between the imaging element 121 (similar to theimaging element 51 in FIG. 1) and the subject surface 31 is a distanced1, the detection signal levels DA, DB, and DC in the pixels Pc, Pb, andPa on the imaging element 121 are expressed by the same equations asequations (1) to (3) described above.DA=α1×a+β1×b+γ1×c  (1)DB=α2×a+β2×b+γ2×c  (2)DC=α3×a+β3×b+γ3×c  (3)

Furthermore, as illustrated in a lower left part of FIG. 13, also in acase where a subject surface 31′ at a subject distance d2 from theimaging element 121 larger than the distance d1 by d, that is, thesubject surface 31′ on a back side of the subject surface 31 as seenfrom the imaging element 121 is considered, the detection signal levelsin the pixels Pc, Pb, and Pa on the imaging element 121 are similar atthe detection signal levels DA, DB, and DC as illustrated in a centralportion in the lower stage of FIG. 13.

However, in this case, light beams of light intensities a′, b′, and c′from point light sources PA′, PB′, and PC′ on the subject surface 31′are received by the respective pixels of the imaging element 121.Furthermore, since the incident angles of the light beams of the lightintensities a′, b′, and c′ on the imaging element 121 are different(change), different coefficient set groups are required. Accordingly,the detection signal levels DA, DB, and DC in the pixels Pa, Pb, and Pc,respectively, are expressed by, for example, following equations (4) to(6).DA=α11×a′+β11×b′+γ11×c′  (4)DB=α12×a′+β12×b′+γ12×c′  (5)DC=α13×a′+β13×b′+γ13×c′  (6)

Here, the coefficient set group including the coefficient sets α11, β11,and γ11, coefficient sets α12, β12, and γ12, and coefficient sets α13,β13, and γ13 is the coefficient set group for the subject surface 31′corresponding to the coefficient sets α1, β1, and γ1, coefficient setsα2, β2, and γ2, and coefficient sets α3, β3, and γ3 for the subjectsurface 31.

Accordingly, by solving the simultaneous equations including equations(4) to (6) using the coefficient set groups α11, β11, γ11, α12, β12,γ12, α13, β13, and γ13 set in advance, it is possible to obtain thelight intensities a′, b′, and c′ of the light beams from the point lightsources PA′, PB′, and PC′ of the subject surface 31′ as illustrated in alower right part of FIG. 13 in a manner similar to that in a case ofobtaining the light intensities a, b, and c of the light beams from thepoint light sources PA, PB, and PC of the subject surface 31. As aresult, it becomes possible to restore a restored image of the subjectsurface 31′.

Therefore, in the imaging device 101 in FIG. 2, by preparing thecoefficient set group for each distance (subject distance) from theimaging element 121 to the subject surface in advance, creating thesimultaneous equations while switching the coefficient set group foreach subject distance, and solving the created simultaneous equations,it is possible to obtain the restored images of the subject surfaces atvarious subject distances on the basis of one detection image. Forexample, by imaging and recording the detection image once, andthereafter switching the coefficient set group according to the distanceto the subject surface by using the recorded detection image to restorethe restored image, it is possible to generate the restored image of thesubject surface at an arbitrary subject distance.

Furthermore, in a case where the subject distance and an angle of viewmay be specified, it is also possible to generate the restored image byusing the detection signal of the pixel having the incident angledirectivity suitable for the imaging of the subject surfacecorresponding to the specified subject distance and angle of viewwithout using all the pixels. Therefore, the restored image may begenerated by using the detection signal of the pixel suitable for theimaging of the subject surface corresponding to the specified subjectdistance and angle of view.

For example, the pixel 121 a that is shielded by the light-shieldingfilm 121 b by a width d1 from each end of four sides as illustrated inan upper stage of FIG. 14 and a pixel 121 a′ that is shielded by thelight-shielding film 121 b by a width d2 (>d1) from each end of foursides as illustrated in a lower stage of FIG. 14 are considered.

FIG. 15 illustrates an example of the incident angle of the incidentlight from the subject surface 31 to a central position C1 of theimaging element 121. Note that, FIG. 15 illustrates the example of theincident angle of the incident light in the horizontal direction, butthis is substantially similar also in the vertical direction.Furthermore, in a right part of FIG. 15, the pixels 121 a and 121 a′ inFIG. 14 are illustrated.

For example, in a case where the pixel 121 a in FIG. 14 is arranged inthe central position C1 of the imaging element 121, a range of theincident angle of the incident light on the pixel 121 a from the subjectsurface 31 is an angle A1 as illustrated in a left part of FIG. 15.Accordingly, the pixel 121 a may receive the incident light by a widthW1 in the horizontal direction of the subject surface 31.

In contrast, in a case where the pixel 121 a′ in FIG. 14 is arranged inthe central position C1 of the imaging element 121, the pixel 121 a′ hasa wider light-shielding range than the pixel 121 a, so that the range ofthe incident angle of the incident light on the pixel 121 a′ from thesubject surface 31 is an angle A2 (<A1) as illustrated in the left partof FIG. 15. Accordingly, the pixel 121 a′ may receive the incident lightby a width W2 (<W1) in the horizontal direction of the subject surface31.

That is, the pixel 121 a having a narrow light-shielding range is awide-angle pixel suitable for imaging a wide range on the subjectsurface 31, whereas the pixel 121 a′ having a wide light-shielding rangeis a narrow-angle pixel suitable for imaging a narrow range on thesubject surface 31. Note that, the wide-angle pixel and the narrow-anglepixel here are expressions that compare both the pixels 121 a and 121 a′in FIG. 14, and are not limited when comparing pixels of other angles ofview.

Therefore, for example, the pixel 121 a is used to restore an image I1in FIG. 14. The image I1 is the image having an angle of view SQ1corresponding to the subject width W1 including an entire person H101 asthe subject in an upper stage of FIG. 16. In contrast, for example, thepixel 121 a′ is used to restore an image 12 in FIG. 14. The image 12 isthe image having an angle of view SQ2 corresponding to the subject widthW2 in which a periphery of a face of the person H101 in the upper stageof FIG. 16 is zoomed up.

Furthermore, for example, as illustrated in a lower stage of FIG. 16, itis considered to arrange a predetermined number of pixels 121 a in FIG.14 in a range ZA enclosed by a dotted line of the imaging elements 121and arrange a predetermined number of pixels 121 a′ in a range ZBenclosed by a dashed-dotted line. Then, for example, when restoring theimage of the angle of view SQ1 corresponding to the subject width W1,the image of the angle of view SQ1 may be appropriately restored byusing the detection signal of each pixel 121 a in the range ZA. On theother hand, when restoring the image of the angle of view SQ2corresponding to the subject width W2, the image of the angle of viewSQ2 may be appropriately restored by using the detection signal of eachpixel 121 a′ in the range ZB.

Note that, since the angle of view SQ2 is narrower than the angle ofview SQ1, in a case of restoring the images of the angle of view SQ2 andthe angle of view SQ1 with the same number of pixels, it is possible toobtain the restored image with a higher image quality when restoring theimage of the angle of view SQ2 than when restoring the image of theangle of view SQ1.

That is, in a case where it is considered to obtain the restored imageusing the same number of pixels, it is possible to obtain the restoredimage with a higher image quality when restoring the image with thenarrower angle of view.

For example, a right part of FIG. 17 illustrates a configuration examplein the range ZA of the imaging element 121 in FIG. 16. A left part ofFIG. 17 illustrates a configuration example of the pixel 121 a in therange ZA.

In FIG. 17, a range in black represents the light-shielding film 121 b,and the light-shielding range of each pixel 121 a is determined, forexample, according to rules illustrated in the left part of FIG. 17.

A main light-shielding portion 2101 in the left part of FIG. 17 (blackpart in the left part of FIG. 17) is a range that is shielded in commonin each pixel 121 a. Specifically, the main light-shielding portion 2101has a range of a width dx1 from left and right sides of the pixel 121 ainto the pixel 121 a, and a range of a height dy1 from upper and lowersides of the pixel 121 a into the pixel 121 a, respectively. Then, ineach pixel 121 a, a rectangular opening Z111 that is not shielded by thelight-shielding film 121 b is provided in a range Z102 inside the mainlight-shielding portion 2101. Accordingly, in each pixel 121 a, a rangeother than the opening Z111 is shielded by the light-shielding film 121b.

Here, the openings Z111 of the respective pixels 121 a are regularlyarranged. Specifically, a position in the horizontal direction of theopening Z111 in each pixel 121 a is the same in the pixels 121 a in thesame column in the vertical direction. Furthermore, a position in thevertical direction of the opening Z111 in each pixel 121 a is the samein the pixels 121 a in the same row in the horizontal direction.

On the other hand, the position in the horizontal direction of theopening Z111 in each pixel 121 a is shifted at a predetermined intervalaccording to the position in the horizontal direction of the pixel 121a. That is, as the position of the pixel 121 a advances rightward, aleft side of the opening 2111 moves to a position shifted rightward bywidths dx1, dx2, . . . , and dxn from the left side of the pixel 121 a.An interval between the widths dx1 and dx2, an interval between thewidths dx2 and dx3, . . . , and an interval between the widths dxn−1 anddxn is a value obtained by dividing a length obtained by subtracting thewidth of the opening Z111 from the width in the horizontal direction ofthe range Z102 by the number of pixels n−1 in the horizontal direction.

Furthermore, the position in the vertical direction of the opening Z111in each pixel 121 a is shifted at a predetermined interval according tothe position in the vertical direction of the pixel 121 a. That is, asthe position of the pixel 121 a advances downward, an upper side of theopening Z111 moves to a position shifted downward by widths dy1, dy2, .. . , and dyn from the upper side of the pixel 121 a. An intervalbetween the heights dy1 and dy2, an interval between the heights dy2 anddy3, . . . , and an interval between the heights dyn−1 and dyn is avalue obtained by dividing a length obtained by subtracting the heightof the opening Z111 from the height in the vertical direction of therange Z102 by the number of pixels m−1 in the vertical direction.

A right part of FIG. 18 illustrates a configuration example within therange ZB of the imaging element 121 in FIG. 16. A left part of FIG. 18illustrates a configuration example of the pixel 121 a′ in the range ZB.

In FIG. 18, a range in black represents the light-shielding film 121 b′,and the light-shielding range of each pixel 121 a′ is determined, forexample, according to rules illustrated in the left part of FIG. 18.

A main light-shielding portion Z151 in the left part of FIG. 18 (blackpart in the left part of FIG. 18) is a range that is shielded in commonin each pixel 121 a′. Specifically, the main light-shielding portion2151 has a range of a width dx1′ from left and right sides of the pixel121 a′ into the pixel 121 a′, and a range of a height dy1′ from upperand lower sides of the pixel 121 a′ into the pixel 121 a′, respectively.Then, in each pixel 121 a′, a rectangular opening Z161 that is notshielded by the light-shielding film 121 b′ is provided in a range Z152inside the main light-shielding portion Z151. Accordingly, in each pixel121 a′, a range other than the opening Z161 is shielded by thelight-shielding film 121 b′.

Here, the openings Z161 of the respective pixels 121 a′ are regularlyarranged in a manner similar to that of the openings Z111 of therespective pixels 121 a in FIG. 17. Specifically, a position in thehorizontal direction of the opening Z161 in each pixel 121 a′ is thesame in the pixels 121 a′ in the same column in the vertical direction.Furthermore, a position in the vertical direction of the opening Z161 ineach pixel 121 a′ is the same in the pixels 121 a′ in the same row inthe horizontal direction.

On the other hand, the position in the horizontal direction of theopening Z161 in each pixel 121 a′ is shifted at a predetermined intervalaccording to the position in the horizontal direction of the pixel 121a′. That is, as the position of the pixel 121 a′ advances rightward, aleft side of the opening Z161 moves to a position shifted rightward bywidths dx1′, dx2′, . . . , and dxn′ from the left side of the pixel 121a′. An interval between the widths dx1′ and dx2′, an interval betweenthe widths dx2′ and dx3′, . . . , and an interval between the widthsdxn−1′ and dxn′ is a value obtained by dividing a length obtained bysubtracting the width of the opening Z161 from the width in thehorizontal direction of the range Z152 by the number of pixels n−1 inthe horizontal direction.

Furthermore, the position in the vertical direction of the opening Z161in each pixel 121 a′ is shifted at a predetermined interval according tothe position in the vertical direction of the pixel 121 a′. That is, asa position of the pixel 121 a′ advances downward, an upper side of theopening Z161 moves to a position shifted downward by widths dy1′, dy2′,. . . , and dyn′ from the upper side of the pixel 121 a′. An intervalbetween the heights dy1′ and dy2′, an interval between the heights dy2′and dy3′, . . . , and an interval between the heights dyn−1′ and dyn′ isa value obtained by dividing a length obtained by subtracting the heightof the opening Z161 from the height in the vertical direction of therange Z152 by the number of pixels m−1 in the vertical direction.

Here, the length obtained by subtracting the width of the opening Z111from the width in the horizontal direction of the range Z102 of thepixel 121 a in FIG. 17 is larger than the width obtained by subtractingthe width of the opening Z161 from the width in the horizontal directionof the range Z152 of the pixel 121 a′ in FIG. 18. Accordingly, a changeinterval between the widths dx1, dx2, . . . , and dxn in FIG. 17 islarger than the change interval between the widths dx1′, dx2′, . . . ,and dxn′ in FIG. 18.

Furthermore, the length obtained by subtracting the height of theopening Z111 from the height in the vertical direction of the range Z102of the pixel 121 a in FIG. 17 is larger than the length obtained bysubtracting the height of the opening Z161 from the height in thevertical direction of the range Z152 of the pixel 121 a′ in FIG. 18.Accordingly, a change interval between the heights dy1, dy2, . . . , anddyn in FIG. 17 is larger than the change interval between the heightsdy1′, dy2′, and dyn′ in FIG. 18.

As described above, the change interval of the positions in thehorizontal and vertical directions of the opening Z111 of thelight-shielding film 121 b of each pixel 121 a in FIG. 17 is differentfrom the change interval of the positions in the horizontal and verticaldirections of the opening Z161 of the light-shielding film 121 b′ ofeach pixel 121 a′ in FIG. 18. Then, this difference in interval is thedifference in subject resolution (angular resolution) in the restoredimage. That is, the change interval of the positions in the horizontaland vertical directions of the opening Z161 of the light-shielding film121 b′ of each pixel 121 a′ in FIG. 18 is narrower than the changeinterval of the positions in the horizontal and vertical directions ofthe opening Z111 of the light-shielding film 121 b of each pixel 121 ain FIG. 17. Accordingly, the restored image restored by using thedetection signal of each pixel 121 a′ in FIG. 18 has higher subjectresolution and higher image quality than the restored image restored byusing the detection signal of each pixel 121 a in FIG. 17.

In this manner, by changing the combination of the light-shielding rangeof the main light-shielding portion and the opening range of theopening, the imaging element 121 including pixels having various anglesof view (having various incident angle directivities) may be realized.

Note that, although the example in which the pixels 121 a and the pixels121 a′ are arranged separately in the range ZA and the range ZB,respectively, is described above, this is for the sake of simplicity,and the pixels 121 a corresponding to different angles of view aredesirably mixedly arranged in the same region.

For example, as illustrated in FIG. 19, four pixels each including twopixels×two pixels indicated by a dotted line are made one unit U, andeach unit U includes a wide-angle pixel 121 a-W, a medium-angle pixel121 a-M, a narrow-angle pixel 121 a-N, and an extremely narrow-anglepixel 121 a-AN.

In this case, for example, in a case where the number of all the pixels121 a is X, it becomes possible to restore the restored image using thedetection images of X/4 pixels for each of the four types of viewangles. At that time, four types of coefficient set groups different foreach angle of view are used, and the restored images having differentangles of view are restored by four different simultaneous equations.

Therefore, by restoring the restored image using the detection imageobtained from the pixel suitable for imaging the angle of view of therestored image to be restored, it becomes possible to obtain anappropriate restored image corresponding to the four types of angles ofview.

Furthermore, it is also possible to interpolate to generate images ofthe angle of view between the four types of angles of view and the angleof view around the same from the images of the four types of angles ofview, and realize pseudo optical zooming by seamlessly generating theimages of the various angles of view.

Note that, for example, in a case where the image having the wide angleof view is obtained as the restored image, all the wide-angle pixels maybe used, or a part of the wide-angle pixels may be used. Furthermore,for example, in a case where the image having the narrow angle of viewis obtained as the restored image, all the narrow-angle pixels may beused, or a part of the narrow-angle pixels may be used.

<Imaging Processing by Imaging Device 101>

Next, imaging processing by the imaging device 101 in FIG. 2 isdescribed with reference to a flowchart in FIG. 20.

At step S1, the imaging element 121 images the subject. Therefore, thedetection signal indicating the detection signal level corresponding tothe amount of incident light from the subject is output from each pixel121 a of the imaging element 121 having different incident angledirectivities, and the imaging element 121 supplies the detection imageincluding the detection signal of each pixel 121 a to the restorationunit 122.

At step S2, the restoration unit 122 obtains the coefficient used forthe image restoration. Specifically, the restoration unit 122 sets thedistance to the subject surface 31 to be restored, that is, the subjectdistance. Note that, an arbitrary method may be adopted as a method ofsetting the subject distance. For example, the restoration unit 122 setsthe subject distance input by the user via the input unit 124 or thesubject distance detected by the detection unit 125 as the distance tothe subject surface 31 to be restored.

Next, the restoration unit 122 reads the coefficient set groupassociated with the set subject distance from the storage unit 128.

At step S3, the restoration unit 122 restores the image using thedetection image and the coefficient. Specifically, the restoration unit122 uses the detection signal level of each pixel of the detection imageand the coefficient set group obtained in the processing at step S2 tocreate the simultaneous equations described with reference to equations(1) to (3) or equations (4) to (6) described above. Next, therestoration unit 122 calculates the light intensity of each point lightsource on the subject surface 31 corresponding to the set subjectdistance by solving the created simultaneous equations. Then, byarranging the pixels having the pixel values according to the calculatedlight intensities according to the arrangement of the respective pointlight sources on the subject surface 31, the restoration unit 122generates the restored image formed as the image of the subject.

At step S4, the imaging device 101 performs various types of processingon the restored image. For example, the restoration unit 122 performsdemosaic processing, y correction, white balance adjustment, conversionprocessing to a predetermined compression format and the like on therestored image as necessary. Furthermore, the restoration unit 122supplies the restored image to the display unit 127 and allows the sameto display the image, supplies the restored image to therecording/playback unit 129 and allows the same to record the image onthe recording medium 130, or outputs the restored image to anotherdevice via the communication unit 131 as necessary, for example.

Thereafter, the imaging processing ends.

Note that, in the description above, the example of restoring therestored image from the detection image using the imaging element 121and the coefficient set group associated with the subject distance isdescribed; however, for example, it is also possible to further preparethe coefficient set group corresponding to the angle of view of therestored image as described above in addition to the subject distanceand restore the restored image by using the coefficient set groupaccording to the subject distance and the angle of view. Note that, theresolution with respect to the subject distance and the angle of viewdepends on the number of prepared coefficient set groups.

Furthermore, in the description of the processing using the flowchart inFIG. 20, the example of using the detection signals of all the pixelsincluded in the detection image is described; however, it is alsopossible to generate the detection image including the detection signalof the pixel having the incident angle directivity corresponding to thespecified subject distance and angle of view among the pixels formingthe imaging element 121 and restore the restored image by using thesame. By such processing, it becomes possible to restore the restoredimage by the detection image suitable for the subject distance and theangle of view of the restored image to be obtained, and restorationaccuracy and image quality of the restored image are improved. That is,in a case where the image corresponding to the specified subjectdistance and angle of view is the image corresponding to the angle ofview SQ1 in FIG. 16, for example, by selecting the pixels 121 a havingthe incident angle directivity corresponding to the angle of view SQ1and restoring the restored image with the detection image obtained fromthem, it becomes possible to restore the image of the angle of view SQ1with high accuracy.

By the processing described above, it becomes possible to realize theimaging device 101 having the imaging element 121 in which each pixelhas incident angle directivity as an indispensable component.

As a result, the imaging lens, the pinhole, and the optical filterdisclosed in the above-described Patent Document and the like are notnecessary, so that the degree of freedom in designing the device may beimproved, and an optical element formed separately from the imagingelement 121 and assumed to be mounted together with the imaging element121 in a stage of forming the imaging device becomes not necessary, sothat the device may be made compact in the incident angle of theincident light and a manufacturing cost may be decreased. Furthermore, alens corresponding to an imaging lens for forming an optical image suchas a focus lens becomes unnecessary. However, a zoom lens that changesmagnification may be provided.

Note that, in the description above, the processing of restoring therestored image corresponding to the predetermined subject distanceimmediately after the detection image is captured is described; however,for example, it is also possible to restore the restored image by usingthe detection image at a desired timing after recording the detectionimage on the recording medium 130 or output the same to another devicevia the communication unit 131 without performing the restoringprocessing immediately. In this case, the restoration of the restoredimage may be performed by the imaging device 101 or another device. Inthis case, for example, it is possible to obtain the restored image forthe subject surface of arbitrary subject distance and angle of view byobtaining the restored image by solving the simultaneous equationscreated by using the coefficient set group according to arbitrarysubject distance and angle of view, thereby realizing refocusing and thelike.

For example, in a case where the imaging device including the imaginglens and the conventional imaging element is used, in order to obtainthe image with various focal distances and angles of view, it isnecessary to image while variously changing the focal distance and angleof view. On the other hand, in the imaging device 101, it is possible torestore the restored image of arbitrary subject distance and angle ofview by switching the coefficient set group in this manner, so thatprocessing of repeatedly imaging while variously changing the focaldistance (that is, the subject distance) and the angle of view is notnecessary.

In this case, for example, the user may also obtain the restored imageof the desired subject distance and angle of view while allowing thedisplay unit 127 to display the restored images that are restored whilechanging the coefficient set groups corresponding to the differentsubject distances and angles of view.

Note that, in a case of recording the detection image, when the subjectdistance and angle of view at the time of restoration are determined,the metadata used for restoration may be associated with the detectionimage. For example, by assigning the metadata to image data includingthe detection image, assigning the same ID to the detection image andthe metadata, or recording the detection image and the metadata on thesame recording medium 130, the detection image and the metadata areassociated with each other.

Note that, in a case where the same ID is assigned to the detectionimage and the metadata, it is possible to record the detection image andthe metadata on different recording media or individually output themfrom the imaging device 101.

Furthermore, the metadata may include the coefficient set group used forrestoration or not. In the latter case, for example, the subjectdistance and angle of view at the time of restoration are included inthe metadata, and the coefficient set group corresponding to the subjectdistance and angle of view is obtained from the storage unit 128 and thelike at the time of restoration.

Moreover, in a case where the restored image is restored immediately atthe time of imaging, for example, an image to be recorded or externallyoutput may be selected from the detection image and the restored image.For example, it is possible to record or externally output both theimages or record or externally output only one of the images.

Furthermore, in a case of capturing a moving image also, it is possibleto select whether or not to restore the restored image at the time ofimaging, or to select the image to be recorded or externally output. Forexample, it is possible to immediately restore the restored image ofeach frame, and record or externally output both or one of the restoredimage and the detection image before restoration while capturing themoving image. In this case, it is also possible to display the restoredimage of each frame as a through image at the time of imaging.Alternatively, for example, it is possible to record or externallyoutput the detection image of each frame without performing restorationprocessing at the time of imaging.

Moreover, when capturing the moving image, for example, it is possibleto select whether or not to restore the restored image, and select theimage to be recorded or externally output for each frame. For example,it is possible to switch whether or not to restore the restored imagefor each frame. Furthermore, for example, it is possible to individuallyswitch whether or not to record the detection image and whether or notto record the restored image for each frame. Furthermore, for example,it is also possible to record the detection images of all the frameswhile assigning the metadata to the detection image of a useful framethat may be used later.

Furthermore, it is also possible to realize an autofocus function as isthe case with the imaging device using the imaging lens. For example,the autofocus function may be realized by determining the optimumsubject distance by a hill-climbing method similar to a contrast autofocus (AF) method on the basis of the restored image.

Moreover, it is possible to generate the restored image using thedetection image captured by the imaging element 121 having incidentangle directivities in a pixel unit as compared to the imaging deviceand the like including the optical filter disclosed in above-describedPatent Document and the like and the conventional imaging element, sothat it becomes possible to realize an increase in pixels or obtain therestored image with high resolution and high angular resolution. Incontrast, in the imaging device including the optical filter and theconventional imaging element, it is difficult to realize the highresolution restored image and the like because it is difficult tominiaturize the optical filter even if the pixels are miniaturized.

Furthermore, in the imaging device 101 of the present disclosure, theimaging element 121 is the indispensable configuration, and does notrequire, for example, the optical filter and the like disclosed inPatent Document and the like described above, so that the optical filteris not bent by heat due to a high-temperature usage environment, and itis possible to realize the imaging device with high environmentalresistance.

Moreover, the imaging device 101 of the present disclosure does notrequire the imaging lens, pinhole, and optical filter disclosed inPatent Document and the like described above, so that it becomespossible to improve the degree of freedom in designing a configurationhaving an imaging function.

<Method of Reducing Processing Load>

By the way, in a case where the light-shielding range (that is, theincident angle directivity) of the light-shielding film 121 b of eachpixel 121 a of the imaging element 121 has randomness, as disorder of adifference in the light-shielding range is larger, a load of theprocessing by the restoration unit 122 is larger. Therefore, it ispossible to reduce the disorder, thereby reducing the processing road bymaking a part of the change in the light-shielding range of thelight-shielding film 121 b of each pixel 121 a regular.

For example, the L-shaped light-shielding film 121 b obtained bycombining the longitudinal band-type and the lateral band-type isformed, and the lateral band-type light-shielding films 121 b having thesame width are combined in a predetermined column direction and thelongitudinal band-type light-shielding films 121 b having the sameheight are combined in a predetermined row direction. Therefore, thelight-shielding range of the light-shielding film 121 b of each pixel121 a changes randomly in a pixel unit while having regularity in thecolumn direction and the row direction. As a result, a difference in thelight-shielding range of the light-shielding film 121 b of each pixel121 a, that is, the disorder in the difference of the incident angledirectivity may be reduced, and the processing load of the restorationunit 122 may be reduced.

Specifically, for example, as illustrated in an imaging element 121″ inFIG. 21, lateral band-type light-shielding films 121 b having the samewidth X0 are used for the pixels in the same column indicated by a rangeZ130, and the longitudinal band-type light-shielding film 121 b havingthe same height Y0 are used for the pixels in the same row indicated bya range Z150. As a result, for the pixel 121 a specified by each row andcolumn, the L-shaped light-shielding film 121 b obtained by combiningthem is used.

Similarly, the lateral band-type light-shielding film 121 b having thesame width X1 are used for the pixels in the same column indicated by arange Z131 adjacent to the range Z130, and the longitudinal band-typelight-shielding film 121 b having the same height Y1 are used for thepixels in the same row indicated by a range Z151 adjacent to the rangeZ150. As a result, for the pixel 121 a specified by each row and column,the L-shaped light-shielding film 121 b obtained by combining them isused.

Moreover, the lateral band-type light-shielding film 121 b having thesame width X2 are used for the pixels in the same column indicated by arange Z132 adjacent to the range Z131, and the longitudinal band-typelight-shielding film 121 b having the same height Y2 are used for thepixels in the same row indicated by a range Z152 adjacent to the rangeZ151. As a result, for the pixel 121 a specified by each row and column,the L-shaped light-shielding film 121 b obtained by combining them isused.

By doing so, it is possible to set the range of the light-shielding filmto different values in a pixel unit while allowing the width andposition in the horizontal direction and the height and the position inthe vertical direction of the light-shielding film 121 b to haveregularity, so that it is possible to control the disorder in the changein the incident angle directivity. As a result, it becomes possible toreduce patterns of the coefficient sets and reduce the processing loadof arithmetic processing in the restoration unit 122.

In further detail, as illustrated in an upper right part of FIG. 22, ina case of obtaining a restored image of N×N pixels from a detectionimage Pic of N pixels×N pixels, a relationship illustrated in a leftpart of FIG. 22 is established by a vector X having pixel values of therespective pixels of the restored image of (N×N) rows×one column aselements, a vector Y having pixel values of the respective pixels of thedetection image of (N×N) rows×one column as elements, and a matrix A of(N×N) rows×(N×N) columns including the coefficient set group.

That is, FIG. 22 illustrates that a result obtained by multiplying therespective elements of the matrix A of (N×N) rows×(N×N) columnsincluding the coefficient set group by the vector X of (N×N) rows×onecolumn representing the restored image is the vector Y of (N×N) rows×onecolumn representing the detection image. Then, from this relationship,for example, the simultaneous equations corresponding to equations (1)to (3) or equations (4) to (6) described above are formed.

Note that, FIG. 22 illustrates that each element of the first columnindicated by a range 2201 of the matrix A corresponds to the element ofthe first row of the vector X, and each element of the N×N-th columnindicated by a range 2202 of the matrix A corresponds to the element ofthe N×N-th row of the vector X.

Note that, in a case of using the pinhole, and in a case of using acondensing function for allowing the incident light incident in the samedirection such as the imaging lens to be incident on both adjacent pixeloutput units, a relationship between the position of each pixel and theincident angle of the light is uniquely determined, so that the matrix Ais a diagonal matrix in which all rightward falling diagonal componentsare one. On the other hand, in a case where neither the pinhole nor theimaging lens is used as in the imaging device 101 in FIG. 2, therelationship between the position of each pixel and the incident angleof light is not uniquely determined, so that the matrix A is not thediagonal matrix.

In other words, the restored image may be obtained by solving thesimultaneous equations based on a determinant illustrated in FIG. 22 andobtaining each element of the vector X.

By the way, in general, the determinant in FIG. 22 is transformed asillustrated in FIG. 23 by multiplying both sides by an inverse matrixA⁻¹ of the matrix A from the left, and each element of the vector Xbeing the detection image is obtained by multiplying the vector Y of thedetection image by an inverse matrix A⁻¹ from the left.

However, in reality, there is a case where the matrix A cannot beobtained correctly, the matrix A cannot be measured correctly, a basisvector of the matrix A nearly linearly dependent and it is not possibleto solve, and each element of the detection image includes noise. Then,for any of these reasons or a combination thereof, the simultaneousequations might not be solved.

Therefore, for example, considering a robust configuration with respectto various errors, following equation (7) using the concept of theregularized least-square method is used.[Mathematical Expression 1]{circumflex over (x)}=min∥A{circumflex over (x)}−y∥ ²+∥γ{circumflex over(x)}∥²  (7)

Here, x with “{circumflex over ( )}” at the top in equation (7)represents the vector X, A represents the matrix A, Y represents thevector Y, γ represents a parameter, ∥A∥ represents a L2 norm(square-root of sum root squares). Here, a first term on the right sideis a norm when minimizing both sides in FIG. 22, and a second term onthe right side is a regularization term.

When this equation (7) is solved for x, following equation (8) isobtained.[Mathematical Expression 2]{circumflex over (x)}=(A ^(t) A+γI)⁻¹ A ^(t) y  (8)

Here, A^(t) represents a transposed matrix of the matrix A, and Irepresents a unit matrix.

However, since the matrix A has an enormous size, a calculation amountand a required memory amount are large.

Therefore, for example, as illustrated in FIG. 24, the matrix A isdecomposed into a matrix AL of N rows×N columns and a matrix AR^(T) of Nrows×N columns, and they are multiplied from former and latter stages ofthe matrix X of N rows×N columns representing the restored image, andthe matrix Y of N rows×N columns representing the detection image isobtained as a result. Therefore, for the matrix A of the number ofelements (N×N)×(N×N), the matrices AL and AR^(T) having the number ofelements (N×N) are obtained, and the number of elements in each matrixbecomes 1/(N×N). As a result, a calculation amount and the requiredmemory amount may be reduced.

The determinant illustrated in FIG. 24 is realized, for example, bymaking the matrix in parentheses in equation (8) the matrix AL andmaking the inverse matrix of the transposed matrix of the matrix A thematrix AR^(T).

In the calculation illustrated in FIG. 24, as illustrated in FIG. 25, anelement group 2222 is obtained by multiplying an element of interest Xpin the matrix X by each element group Z221 of the corresponding columnof the matrix AL. Moreover, a two-dimensional response Z224corresponding to the element of interest Xp is obtained by multiplyingthe element group Z222 by the elements in the row corresponding to theelement of interest Xp of the matrix AR^(T). Then, the matrix Y isobtained by integrating the two-dimensional responses Z224 correspondingto all the elements of the matrix X.

Therefore, for example, in the element group Z221 of each column of thematrix AL, a coefficient corresponding to the incident angle directivityof the lateral-band type pixel 121 a set to have the same width for eachcolumn of the imaging elements 121 illustrated in FIG. 21 is used.

Similarly, for example, in the element group Z223 of each row of thematrix AR^(T), a coefficient corresponding to the incident angledirectivity of the longitudinal-band type pixel 121 a set to have thesame height for each row of the imaging elements 121 illustrated in FIG.21 is used.

As a result, since it becomes possible to reduce the matrix used whenrestoring the restored image on the basis of the detection image, thecalculation amount may be reduced, a processing speed may be improved,and power consumption for the calculation may be reduced. Furthermore,since the matrix may be reduced, a capacity of the memory used for thecalculation may be reduced, and a device cost may be reduced.

Note that, although FIG. 21 illustrates the example of changing thelight-shielding range (light-receiving range) in a pixel unit whileproviding predetermined regularity in the horizontal direction and thevertical direction, in the present disclosure, the light-shielding range(light-receiving range) not completely randomly set in a pixel unit butrandomly set to a certain degree in this manner is also considered to berandomly set. In other words, in the present disclosure, not only a casewhere the light-shielding range (light-receiving range) is setcompletely at random in a unit of pixel, but also a case at random to acertain degree (for example, a case where a part of all the pixels has arange with regularity but other range is at random), or a caseapparently not regular to a certain degree (a case of arrangement inwhich it is not possible to confirm arrangement according to theregularity as described with reference to FIG. 21 among all the pixels)are also considered to be random.

3. First Embodiment

Next, a first embodiment of the present disclosure is described withreference to FIGS. 26 to 36.

As described above, an imaging element 121 using a pixel having incidentangle directivity does not require an imaging lens, a pinhole, and anoptical filter disclosed in Patent Document and the like describedabove, so that a degree of freedom in arrangement of respective pixels121 a is high.

Therefore, in the first embodiment, a subject is imaged by a pluralityof imaging devices including the imaging element having the incidentangle directivity, and a restored image is restored using a detectionsignal obtained by each imaging device.

<Configuration Example of Imaging System 301>

FIG. 26 is a block diagram illustrating a configuration example of animaging system 301 according to the first embodiment of the presentdisclosure.

The imaging system 301 includes an imaging unit 311 and a signalprocessing unit 312.

The imaging unit 311 includes one or more n imaging devices 321-1 to321-n. Note that, hereinafter, in a case where it is not necessary todistinguish the imaging devices 321-1 to 321-n from one another, theyare simply referred to as the imaging devices 321.

Each imaging device 321 includes one or more imaging elements 121 havingthe incident angle directivity described above. Each imaging device 321may be discretely arranged in a space, for example, as illustrated inFIG. 27. That is, the respective imaging devices 321 do not necessarilyaligned in the space, and may be arranged at random with a spacetherebetween.

Each imaging device 321 performs short-range wireless communication witha reader/writer 331 of the signal processing unit 312 using, forexample, a technology such as radio frequency identifier (RFID), andtransmits a detection signal set including one or more detection signalsobtained by each imaging element 121 to the reader/writer 331.Furthermore, each imaging device 321 converts an electromagnetic wavetransmitted from the reader/writer 331 into electric power, and isdriven by the converted electric power. In other words, each imagingdevice 321 is driven by an electromotive force by the electromagneticwave transmitted from the reader/writer 331.

The signal processing unit 312 obtains the detection signal set fromeach imaging device 321 by controlling imaging by each imaging device321, and performs restoration processing and the like of the restoredimage using the obtained detection signal set. The signal processingunit 312 includes the reader/writer 331 and a signal processing device332.

As described above, the reader/writer 331 performs the short-rangewireless communication with each imaging device 321 using the technologysuch as RFID, for example, and supplies electric power to each imagingdevice 321.

As illustrated in FIG. 27, the signal processing device 332 performs therestoration processing and the like of the restored image on the basisof the detection signal set received from each imaging device 321 viathe reader/writer 331.

Note that, hereinafter, different alphabets are assigned to the end ofreference signs as an imaging device 321A and an imaging device 321Bdepending on the embodiments of the imaging device 321; they are simplyreferred to as the imaging devices 321 in a case where it is notnecessary to especially distinguish them from one another.

<Configuration Example of Imaging Device 321A>

FIG. 28 is a block diagram illustrating a configuration example of theimaging device 321A as the first embodiment of the imaging device 321 inFIG. 26. Note that, in the drawing, a portion corresponding to that ofthe imaging device 101 in FIG. 2 is assigned with the same referencesign, and the description thereof is omitted as appropriate.

The imaging device 321A includes an imaging element 121, a control unit411, an association unit 412, a storage unit 413, and a communicationunit 414. Furthermore, the control unit 411, the association unit 412,the storage unit 413, and the communication unit 414 form a signalprocessing control unit 401A. Note that, the imaging device 321A doesnot include an imaging lens.

Furthermore, the imaging element 121, the control unit 411, theassociation unit 412, the storage unit 413, and the communication unit414 are connected to one another via a bus B2, and perform transmission,reception and the like of data via the bus B2. Note that, hereinafter,in order to simplify the description, description of the bus B2 in acase where each unit of the imaging device 321A performs thetransmission, reception and the like of the data via the bus B2 isomitted.

The imaging element 121 outputs the detection signal set including oneor more detection signals output from each pixel 121 a to the bus B2.

The control unit 411 includes, for example, various processors andcontrols each unit of the imaging device 321A.

The association unit 412 associates the detection signal set obtained bythe imaging element 121 with metadata corresponding to the detectionsignal set in cooperation with an association unit 504 (FIG. 33) of thesignal processing device 332 or alone.

The storage unit 413 includes one or more storage devices such as a readonly memory (ROM), a random access memory (RAM), and a flash memory, andstores, for example, programs, data and the like used for processing ofthe imaging device 321A. The storage unit 413 stores, for example, an IDfor uniquely identifying the imaging device 321A.

The communication unit 414 performs the short-range wirelesscommunication with the reader/writer 331 and receives the electric powertransmitted from the reader/writer 331 by electromagnetic wave. Thecommunication unit 414 includes a transmission/reception unit 421 and anantenna 422.

The transmission/reception unit 421 transmits/receives data to/from thereader/writer 331 via the antenna 422 by a short-range wirelesscommunication method supporting the reader/writer 331.

The antenna 422 transmits and receives the electromagnetic wave to andfrom an antenna (not illustrated) of the reader/writer 331. Furthermore,the electromagnetic wave received from the reader/writer 331 by theantenna 422 is converted into electric power, and the imaging device321A is driven by the electric power.

FIG. 29 schematically illustrates a configuration example of anappearance of the imaging device 321A. In an upper stage of FIG. 29, aconfiguration example of a front surface of the imaging device 321A isschematically illustrated, and in a lower stage, a configuration exampleof a back surface of the imaging device 321A is schematicallyillustrated.

In the imaging device 321A, rectangular antennas 422 are provided onright and left sides of a vertically long rectangular main body 431 atthe center. The main body 431 includes the imaging element 121, thecontrol unit 411, the association unit 412, the storage unit 413, andthe transmission/reception unit 421. Furthermore, a light-receivingsurface 121A of the imaging element 121 is arranged at substantially thecenter of a surface of the main body 431.

FIGS. 30 to 32 illustrate examples of a pattern of a pixel array unit ofthe imaging element 121 of each imaging device 321A. Note that, a blackportion in FIGS. 30 to 32 indicates a light-shielding range by alight-shielding film 121 b.

FIG. 30 illustrates an example of a case where the imaging element 121of each imaging device 321A includes one pixel 121 a.

In patterns Pt1 to Pt4, color filters 121 d of different colors ofwhite, red, green, or blue are provided, and a right half islight-shielded. Note that, in a case of white, the color filter 121 dmay be a transparent filter or the filter is not required. Therefore, byusing four imaging devices 321A of the patterns Pt1 to Pt4, it ispossible to obtain the detection signals having the same incident angledirectivity and different colors (detection wavelengths). Then, forexample, by using a plurality of imaging devices 321A having differentlight-shielding ranges of each color, a plurality of detection signalshaving different incident angle directivities is obtained for eachcolor.

FIG. 31 illustrates an example of a case where the imaging element 121of each imaging device 321A includes three pixels 121 a.

In a pattern Pt11, the respective pixels 121 a are arranged in thehorizontal direction and the respective pixels 121 a are provided withthe color filters 121 d of different colors of red, green, or blue.Furthermore, the right half of each pixel 121 a is light-shielded.Therefore, by using the imaging device 321A of the pattern Pt11, it ispossible to obtain the detection signals having the same incident angledirectivity and different colors (detection wavelengths) at once. Then,for example, by using a plurality of imaging devices 321A having thesame color array as the pattern Pt11 and having differentlight-shielding ranges, a plurality of detection signals havingdifferent incident angle directivities is obtained for each color.

In a pattern Pt12, as is the case with the pattern Pt11, the pixels 121a are arranged in the horizontal direction and the respective pixels 121a includes the color filters 121 d of different colors of red, green, orblue. Furthermore, in the red pixel 121 a and the blue pixel 121 a, theright half is light-shielded, and in the green pixel 121 a, the lefthalf is light-shielded. Therefore, by using the imaging device 321Ahaving the pattern Pt12, detection signals having different incidentangle directivities and colors (detection wavelengths) may be obtainedat a time. Then, for example, by using a plurality of imaging devices321A having the same color array as the pattern Pt12 and havingdifferent light-shielding ranges, a plurality of detection signalshaving different incident angle directivities is obtained for eachcolor.

FIG. 32 illustrates an example in which the imaging element 121 of eachimaging device 321A includes four pixels 121 a.

In a pattern Pt21, the respective pixels 121 a are arranged in thehorizontal direction and the respective pixels 121 a are provided withthe color filters 121 d of different colors of red, green, blue, orwhite. Note that, in a case of white, the color filter 121 d may be atransparent filter or the filter is not required. Furthermore, the righthalf of each pixel 121 a is light-shielded. Therefore, by using theimaging device 321A of the pattern Pt21, it is possible to obtain thedetection signals having the same incident angle directivity anddifferent colors (detection wavelengths) at once. Then, for example, byusing a plurality of imaging devices 321A having the same color array asthe pattern Pt21 and having different light-shielding ranges, aplurality of detection signals having different incident angledirectivities is obtained for each color.

In a pattern Pt22, the respective pixels 121 a are arranged in tworows×two columns and the respective pixels 121 a are provided with thecolor filters 121 d of different colors of red, green, blue, or white.Note that, in a case of white, the color filter 121 d may be atransparent filter or the filter is not required. Furthermore, the righthalf of each pixel 121 a is light-shielded. Therefore, by using theimaging device 321A of the pattern Pt22, it is possible to obtain thedetection signals having the same incident angle directivity anddifferent colors (detection wavelengths) at once. Then, for example, byusing a plurality of imaging devices 321A having the same color array asthe pattern Pt22 and having different light-shielding ranges, aplurality of detection signals having different incident angledirectivities is obtained for each color.

In a pattern Pt23, the respective pixels 121 a are arranged in tworows×two columns, and the color array of the respective pixels 121 a isaccording to the Bayer array. Furthermore, the light-shielding range ofeach pixel 121 a is different. Therefore, by using the imaging device321A having the pattern Pt23, detection signals having differentincident angle directivities and colors (detection wavelengths) may beobtained at a time. Then, for example, by using a plurality of imagingdevices 321A having the same color array as the pattern Pt23 and havingdifferent light-shielding ranges, a plurality of detection signalshaving different incident angle directivities is obtained for eachcolor.

In a pattern Pt24, the respective pixels 121 a are arranged in tworows×two columns, and the respective pixels 121 a are provided with thecolor filters 121 d of the same color (white in this example). Notethat, in a case of white, the color filter 121 d may be a transparentfilter or the filter is not required. Furthermore, the light-shieldingrange of each pixel 121 a is different. Therefore, by using the imagingdevice 321A of the pattern Pt23, it is possible to obtain the detectionsignals having different incident angle directivities and the same color(detection wavelength) at once.

<Configuration Example of Signal Processing Device 332>

FIG. 33 is a block diagram illustrating a configuration example of thesignal processing device 332 in FIG. 26. The signal processing device332 includes a restoration unit 501, a control unit 502, an input unit503, an association unit 504, a display unit 505, a storage unit 506, arecording/playback unit 507, a recording medium 508, and a communicationunit 509.

The restoration unit 501, the control unit 502, the input unit 503, thedisplay unit 505, the storage unit 506, the recording/playback unit 507,and the communication unit 509 are connected to one another via a busB3, and perform transmission, reception and the like of data via the busB3. Note that, hereinafter, in order to simplify the description,description of the bus B3 is omitted in a case where each unit of thesignal processing device 332 performs transmission and reception of thedata and the like via the bus B3.

The restoration unit 501 performs the restoration processing and thelike of the restored image by processing similar to that of therestoration unit 122 of the imaging device 101 in FIG. 2 by using thedetection signal set received from each imaging device 321 via thereader/writer 331. The restoration unit 501 outputs the restored imageto the bus B3.

The control unit 502 includes, for example, various processors andcontrols each unit of the signal processing device 332.

The input unit 503 includes an input device (for example, a key, aswitch, a button, a dial, a touch panel, a remote controller and thelike) for operating the signal processing device 332, inputting dataused for processing and the like. The input unit 503 outputs anoperation signal, the input data and the like to the bus B3.

The association unit 504 associates the detection signal set obtainedfrom each imaging device 321A with the metadata corresponding to eachdetection signal set in cooperation with the association unit 412 of theimaging device 321A or alone.

The display unit 505 includes a display, for example, and displaysvarious types of information (for example, the restored image and thelike). Note that, the display unit 505 may include an audio output unitsuch as a speaker to output audio.

The storage unit 506 includes one or more storage devices such as a ROM,a RAM, and a flash memory, and stores, for example, programs and dataused for processing of the signal processing device 332. For example,the storage unit 506 stores installation information including aposition (for example, latitude, longitude and the like), an attitude(for example, orientation, inclination and the like) and the like ofeach imaging device 321A. Furthermore, the storage unit 506 stores, forexample, a coefficient set group corresponding to the imaging element121 of each imaging device 321A in a current installation state of eachimaging device 321A. This coefficient set group is prepared for eachassumed subject distance and angle of view, for example.

The recording/playback unit 507 records the data on the recording medium508 and plays back (reads out) the data recorded on the recording medium508. For example, the recording/playback unit 507 records the restoredimage on the recording medium 508 or reads out the same from therecording medium 508. Furthermore, for example, the recording/playbackunit 507 records the detection signal set and the corresponding metadataon the recording medium 508 or reads out the same from the recordingmedium 508.

The recording medium 508 includes, for example, any one of a HDD, anSSD, a magnetic disk, an optical disk, a magneto-optical disk, asemiconductor memory or the like, a combination thereof or the like.

The communication unit 509 communicates with other devices (for example,the reader/writer 331, another signal processing device and the like) bya predetermined communication method. Note that, the communicationmethod of the communication unit 509 may be wired or wireless.Furthermore, the communication unit 509 may support a plurality ofcommunication methods.

<Processing of Imaging System 301>

Next, processing of the imaging system 301 is described with referenceto FIGS. 34 to 36.

<Processing of Signal Processing Unit 312>

First, processing of the signal processing unit 312 is described withreference to a flowchart in FIG. 34.

At step S101, the signal processing unit 312 transmits an imagingcommand.

Specifically, the control unit 502 of the signal processing device 332generates the imaging command and codes the generated imaging command.The control unit 502 supplies the coded imaging command to thereader/writer 331 via the communication unit 509.

The reader/writer 331 starts transmitting a carrier wave including theelectromagnetic wave and modulates the carrier wave with the codedimaging command to transmit the imaging command to each imaging device321.

Each imaging device 321 receives the imaging command at step S151 inFIG. 36 to be described later and transmits imaging data at step S153.

At step S102, the signal processing unit 312 receives the imaging datafrom each imaging device 321A.

Specifically, the reader/writer 331 receives the imaging datatransmitted from each imaging device 321 by modulating the carrier waveincluding a reflected wave of the transmitted electromagnetic wave.

The imaging data is transmitted using, for example, a packet illustratedin FIG. 35. Each packet includes a start code, an ID, a detectionsignal, and an end code.

A predetermined code indicating a head of the packet is set as the startcode.

The ID is the ID for identifying the imaging device 321A and the pixel121 a which output the detection signal included in the packet, the IDobtained by combining the ID for identifying each imaging device 321Aand the ID for identifying each pixel 121 a in the imaging element 121,for example. Therefore, the imaging device 321A and the pixel 121 a thatoutput the detection signal included in the packet are specified.

In the detection signal, a value of the detection signal to betransmitted (that is, the detection signal level) is set.

A predetermined code indicating the end of the packet is set as the endcode.

Note that, for example, the ID and the detection signal are coded asnecessary.

For example, one packet is generated for one detection signal, and eachdetection signal is transmitted by a different packet. Accordingly, atleast as many packets as the number of detection signals are transmittedfrom each imaging device 321A. Then, with one or more packets, thedetection signal set including one or more detection signals output fromthe imaging element 121 of each imaging device 321A, and the metadata(for example, ID and the like) corresponding to the detection signal setare transmitted.

Note that, for example, a plurality of detection signals may betransmitted at once by including a plurality of combinations of ID anddetection signal in one packet. In this case, for example, if all thedetection signals are included in one packet to be transmitted, it isalso possible to omit transmission of the ID of each pixel 121 a byfixing the arranging order of the detection signals, and include onlyone ID of the imaging device 321A in one packet.

The reader/writer 331 demodulates each packet of the imaging datareceived from each imaging device 321A and supplies the same to thesignal processing device 332. The restoration unit 501 of the signalprocessing device 332 obtains each demodulated packet via thecommunication unit 509, and decodes the ID and the detection signalincluded in each packet.

Furthermore, the reader/writer 331 stops transmitting the carrier waveincluding the electromagnetic wave after an elapse of a predeterminedtime.

At step S103, the restoration unit 501 obtains a coefficient used forimage restoration. Specifically, the restoration unit 501 sets thesubject distance by the processing similar to that by the restorationunit 122 of the imaging device 101 at step S2 in FIG. 20. Then, therestoration unit 122 reads out the coefficient set group associated withthe set subject distance from the storage unit 506.

At step S104, the restoration unit 501 restores the image using thedetection signal set and the coefficient. That is, the restoration unit501 restores the restored image by using the detection signal setincluded in the imaging data received from each imaging device 321A andthe coefficient set group obtained in the processing at step S103 by theprocessing similar to that by the restoration unit 122 of the imagingdevice 101 at step S3 in FIG. 20.

At step S105, the signal processing device 332 performs various types ofprocessing on the restored image. For example, the restoration unit 501performs demosaic processing, y correction, white balance adjustment,conversion processing to a predetermined compression format and the likeon the restored image as necessary. Furthermore, the restoration unit501 supplies the restored image to the display unit 505 and allows thesame to display the image, supplies the restored image to therecording/playback unit 507 and allows the same to record the image onthe recording medium 508, or outputs the restored image to anotherdevice via the communication unit 509 as necessary, for example.

Thereafter, the processing of the signal processing unit 312 ends.

Next, with reference to a flowchart in FIG. 36, processing executed byeach imaging device 321A corresponding to the processing of the signalprocessing unit 312 in FIG. 34 is described.

At step S151, the imaging device 321A receives the imaging command.Specifically, the antenna 422 of the imaging device 321A receives theelectromagnetic wave of which transmission from the reader/writer 331 isstarted at step S101 in FIG. 34 and converts the receivedelectromagnetic wave into electric power to supply to each unit of theimaging device 321A. Therefore, each unit of the imaging device 321A isactivated.

Next, the transmission/reception unit 421 demodulates theelectromagnetic wave received from the reader/writer 331 via the antenna422, and supplies the coded data obtained as a result to the controlunit 411.

The control unit 411 obtains the imaging command by decoding the codeddata.

At step S152, the imaging element 121 images the subject as in theprocessing at step S1 in FIG. 20. The imaging element 121 supplies thedetection signal set including the detection signals of the respectivepixels 121 a to the association unit 412.

At step S153, the imaging device 321A transmits the imaging data.

Specifically, the association unit 412 generates the packet describedabove with reference to FIG. 35 for each detection signal included inthe detection signal set. Therefore, the ID is assigned to eachdetection signal. The association unit 412 supplies the respectivegenerated packets to the transmission/reception unit 421.

The transmission/reception unit 421 transmits the imaging data to thereader/writer 331 in a packet unit by modulating the carrier waveincluding the reflected wave of the electromagnetic wave received fromthe reader/writer 331 via the antenna 422 with data of each packet, forexample.

Thereafter, the reader/writer 331 stops transmitting the electromagneticwave, so that the supply of the electric power from the reader/writer331 is finished, the imaging device 321A is turned off, and theprocessing ends.

As described above, a plurality of imaging devices 321A images incorporation, so that the imaging device 321A may be made more compact,power consumption may be reduced, and an amount of data communicationrelated to the transmission of the imaging data may be reduced.Furthermore, the imaging device 321A is compact, does not require apower source such as a battery, and transmits the imaging data bywireless communication, so that this has a high degree of freedom ininstallation position. Therefore, an application range of the imagingdevice 321A may be expanded.

For example, by applying the imaging system 301 to an agriculturalmonitoring system, arranging the imaging devices 321A in a plurality oflocations in a farmland, and performing the above-described processing,it is possible to grasp the weather and crop conditions from a remotelocation.

Furthermore, by arranging the compact imaging devices 321A in adistributed manner, it becomes possible to image without damaging thelandscape. For example, by applying the imaging system 301 to a buildingmonitoring system and arranging a plurality of imaging devices 321A in adistributed manner on a building wall and the like, it is possible toimage the surroundings of the building without damaging the landscapeand monitor from a remote place. Furthermore, in this case, there alsois an effect that the presence of the imaging device 321A is hardlynoticed.

Moreover, since the imaging device 321A may be installed on a compactdevice or in a curved portion of the device, a range of applicabledevices is expanded. For example, the imaging device 321A is applicableto various wearable terminals, medical devices such as an endoscopecamera and a fundus camera, a card-type device and the like.

4. Second Embodiment

Next, a second embodiment of the present disclosure is described withreference to FIG. 37.

<Configuration Example of Imaging Device 321B>

In the second embodiment of the present disclosure, an imaging device321B in FIG. 37 that is a second embodiment of the imaging device 321 isused in the imaging system 301 in FIG. 26. Note that, in FIG. 37, thesame reference sign is assigned to a portion corresponding to that ofthe imaging device 321A in FIG. 28 and the description thereof isappropriately omitted. Furthermore, in FIG. 37, power supply lines forsupplying electric power from a power supply unit 451 to each unit ofthe imaging device 321B are not illustrated.

The imaging device 321B is different from the imaging device 321A inthat the power supply unit 451 is provided.

The power supply unit 451 includes, for example, a battery or an AC/DCpower supply, and supplies electric power for driving to each unit ofthe imaging device 321B. Note that, in a case where the power supplyunit 451 is the AC/DC power supply, the power supply unit 451 convertsthe electric power supplied from an external AC power supply into adirect current, and then supplies the electric power to each unit of theimaging device 321B.

In this manner, by providing the power supply unit 451, the imagingdevice 321B may autonomously perform imaging processing and transmitimaging data even when the electric power is not supplied from areader/writer 331.

In contrast, the signal processing device 332 may receive the imagingdata transmitted from each imaging device 321 without transmitting animaging command to each imaging device 321B and restore a restored imageby using the received imaging data.

5. Third Embodiment

Next, a third embodiment of the present disclosure is described withreference to FIG. 38.

<Configuration Example of Imaging Device 321C>

In the third embodiment of the present disclosure, an imaging device321C in FIG. 38 that is a third embodiment of the imaging device 321 isused in the imaging system 301 in FIG. 26. Note that, in FIG. 38, thesame reference sign is assigned to a portion corresponding to that ofthe imaging device 321A in FIG. 28 and the description thereof isappropriately omitted. Furthermore, in FIG. 38, power supply lines forsupplying electric power from a power supply unit 461 to each unit ofthe imaging device 321C are not illustrated.

The imaging device 321C is different from the imaging device 321A inthat the power supply unit 461 is provided.

The power supply unit 461 is a power supply that generates electricpower by solar power generation. The power supply unit 461 includes aphotoelectric conversion unit 471 and a power storage unit 472.

The photoelectric conversion unit 471 includes, for example, aphotodiode, converts received light into charges, and supplies theobtained charges to the power storage unit 472.

The power storage unit 472 accumulates the charges converted by thephotoelectric conversion unit 471 and supplies the electric power by theaccumulated charges to each unit of the imaging device 321C.

Therefore, the imaging device 321C may generate the electric power byitself and operate with the electric power without externally receivingthe electric power supply or replacing a battery.

Note that, for example, the photoelectric conversion unit 471 may beintegrated with an imaging element 121 by forming a photodiode of theimaging element 121 and the photodiode of the photoelectric conversionunit 471 on the same semiconductor substrate. Therefore, the imagingdevice 321C may be made compact and a manufacturing process may besimplified.

6. Fourth Embodiment

Next, a fourth embodiment of the present disclosure is described withreference to FIGS. 39 to 45.

Since an imaging device 321 is compact and has a high degree of freedomin installation position as described above, this is assumed to be usedin an arbitrary place other than a predetermined place. For example, itis assumed that a plurality of imaging devices 321 is disorderlydistributed to be used.

In contrast, when a position and an attitude of the imaging device 321change, an inclination of a light-receiving surface of an imagingelement 121 with respect to a subject surface to be restored changes,and an incident angle of a light beam from each point light source onthe subject surface changes. For example, as illustrated in FIG. 39,when the imaging device 321 moves from a position indicated by a dottedline to a position indicated by a solid line, the incident angle of thelight beam from a point light source P of a subject surface 31 on (thelight-receiving surface of the imaging element 121 of) the imagingdevice 321 changes.

In contrast, as described above, each pixel 121 a of the imaging element121 has incident angle directivity. For example, FIG. 40 illustrates anexample of light-receiving sensitivity to an incident angle θx in thehorizontal direction of a certain pixel 121 a in the imaging element121. In FIG. 40, the incident angle θx in the horizontal direction isplotted along the abscissa, and a detection signal level is plottedalong the ordinate.

For example, in a case where the light beam is incident from the pointlight source P of the subject surface 31 at an incident angle of 0°, ifan inclination in the horizontal direction of the imaging element 121increases by Δθx, the detection signal level increases as illustrated inthe drawing.

Therefore, in a case where the position and attitude of the imagingdevice 321 change, it is necessary to adjust a coefficient used forrestoring a restored image.

Therefore, in the fourth embodiment, the position and inclination ofeach imaging device 321 are detected by each imaging device 321, and asignal processing device 332 obtains the coefficient used for restoringthe restored image on the basis of a detection result, and the restoredimage is restored.

<Configuration Example of Imaging Device 321D>

In the fourth embodiment, an imaging device 321D in FIG. 41 that is afourth embodiment of the imaging device 321 is used in the imagingsystem 301 in FIG. 26. Note that, in FIG. 41, the same reference sign isassigned to a portion corresponding to that of the imaging device 321Ain FIG. 28 and the description thereof is appropriately omitted.

The imaging device 321D is different from the imaging device 321A inthat a signal processing control unit 401D is provided in place of thesignal processing control unit 401A. The signal processing control unit401D is different from the signal processing control unit 401A in that adetection unit 611A is added.

The detection unit 611A includes a position detection unit 621 and aninclination detection unit 622.

The position detection unit 621 includes, for example, a GNSS receiverand the like and detects a current position (for example, an absoluteposition represented by latitude and longitude) of the imaging device321D. The position detection unit 621 outputs the detected positioninformation to a bus B2.

The inclination detection unit 622 includes, for example, anacceleration sensor or an inclination sensor, and detects theinclination of the light-receiving surface of the imaging element 121with respect to a plane perpendicular to a direction of gravity. Forexample, as illustrated in FIG. 42, in a case where the direction ofgravity is a Z-axis, a direction perpendicular to the Z-axis andextending in an east-west direction is an X-axis, and a directionperpendicular to the Z-axis and extending in a north-south direction isa Y-axis, the inclination detection unit 622 detects inclinations Xθ andYθ. The inclination Xθ is the inclination around the Y-axis in an X-Zplane. The inclination Yθ is the inclination around the X-axis in a Y-Zplane. The inclination detection unit 622 outputs detected inclinationinformation to the bus B2.

<Processing of Imaging System 301>

Next, processing of an imaging system 301 is described with reference toFIGS. 43 to 45.

<Processing of Signal Processing Unit 312>

First, processing of a signal processing unit 312 is described withreference to a flowchart in FIG. 43.

At step S201, an imaging command is transmitted as in the processing atstep S101 in FIG. 34.

At step S202, imaging data is received from each imaging device 321D asin the processing at step S102 in FIG. 34. However, a data configurationof a packet of the imaging data is different from that in the processingat step S102 in FIG. 34.

FIG. 44 illustrates an example of the data configuration of the packetof the imaging data used in the fourth embodiment.

The data configuration of the packet in FIG. 44 is different from thedata configuration of the packet in FIG. 35 in that the inclination Xθ,the inclination Yθ, the latitude, and the longitude are added.

As the inclinations Xθ and Yθ, the inclinations Xθ and the inclinationYθ of the light-receiving surface of the imaging element 121 detected bythe inclination detection unit 622 of the imaging device 321D are set,respectively.

As the latitude and longitude, the latitude and longitude of the currentposition of the imaging device 321D detected by the position detectionunit 621 of the imaging device 321D are set, respectively.

At step S203, the restoration unit 501 obtains the coefficient used forimage restoration. Specifically, the restoration unit 501 sets a subjectdistance as in the processing at step S103 in FIG. 34.

Next, the restoration unit 501 calculates, for each imaging device 321D,an amount of change in inclination of the light-receiving surface of theimaging element 121 with respect to the subject surface at the setsubject distance (the subject surface to be restored). That is, therestoration unit 501 calculates the amount of change in the inclinationof the light-receiving surface of the imaging element 121 with respectto the subject surface in association with the movement of each imagingdevice 321D from a predetermined reference position to the currentposition.

Hereinafter, it is assumed that the light-receiving surface of theimaging element 121 and the subject surface face each other in a casewhere the imaging device 321D is installed in the reference position,and are surfaces perpendicular to the direction of gravity. Furthermore,hereinafter, the latitude and longitude of the reference position arereferred to as reference latitude and reference longitude, respectively.Note that, the reference position is set in advance for each imagingdevice 321D, for example.

In this case, in a case where the latitude and longitude and theinclination Xθ and the inclination Yθ of the imaging device 321D changefrom the reference position, a change amount Δθx of the incident angleθx in the X-axis direction and a change amount Δθy of the incident angleθy in the Y-axis direction of the light beam on the light-receivingsurface of the imaging element 121 from each point light source of thesubject surface are expressed by following equations (9) and (10),respectively.Δθx=atan((longitude after movement−reference longitude)/subjectdistance)+Xθ  (9)Δθy=atan((latitude after movement−reference latitude)/subjectdistance)+Yθ  (10)

That is, the incident angle θx in the X-axis direction and the incidentangle θy in the Y-axis direction of the light beam on thelight-receiving surface of the imaging element 121 from each point lightsource of the subject surface increase by Δθx and Δθy, respectively.

Note that, hereinafter, the change amount Δθx of the incident angle θxis referred to as an incident angle change amount Δθx, and the changeamount Δθy of the incident angle θy is referred to as an incident anglechange amount Δθy.

Therefore, the restoration unit 501 calculates the incident angle changeamount Δθx and the incident angle change amount Δθy of each imagingdevice 321D with respect to the subject surface 31 at the set subjectdistance on the basis of the latitude, longitude, inclination Xθ, andinclination Yθ of each imaging device 321D. The incident angle changeamount Δθx and the incident angle change amount Δθy are the changeamounts in the incident angle θx and the incident angle θy withreference to a case where each imaging device 321D is installed in apredetermined reference position.

At that time, for example, the storage unit 506 stores not only acoefficient set group for the subject surface at each subject distancein a case where each imaging device 321D is installed in the referenceposition but also the coefficient set group for each of the incidentangle change amount Δθx and the incident angle change amount Δθy foreach subject surface. Then, the restoration unit 501 reads out thecoefficient set group corresponding to the set subject distance and thecalculated incident angle change amount Δθx and incident angle changeamount Δθy from the storage unit 506 as the coefficient set groupcorresponding to each imaging device 321D.

Alternatively, for example, in a case of obtaining the coefficient byusing the characteristics of the weight Wx and the weight Wy describedabove with reference to FIG. 9, the restoration unit 501 may obtain thecoefficient on the basis of the weight Wx and the weight Wy for theincident angle obtained by adding the calculated incident angle changeamount Δθx and the incident angle change amount Δθy to the incidentangle ex and the incident angle θy of the incident light from each pointlight source of the subject surface in a case where the imaging device321D is installed in the reference position.

At step S204, as in the processing at step S104 in FIG. 34, imagerestoration is performed using a detection signal set and thecoefficient.

At step S205, as in the processing at step S105 in FIG. 34, varioustypes of processing are performed on the restored image.

Thereafter, the processing of the signal processing unit 312 ends.

<Processing of Imaging Device 321D>

Next, with reference to a flowchart in FIG. 45, processing executed byeach imaging device 321D corresponding to the processing of the signalprocessing unit 312 in FIG. 43 is described.

At step S251, the imaging command is received as in the processing atstep S151 in FIG. 36.

At step S252, the subject is imaged as in the processing at step S152 inFIG. 36.

At step S253, the detection unit 611A detects the position and attitude.Specifically, the position detection unit 621 detects the latitude andlongitude of the current position of the imaging device 321D and outputsthe detected position information to the bus B2. The inclinationdetection unit 622 detects the inclination Xθ and the inclination Yθ ofthe light-receiving surface of the imaging element 121 of the imagingdevice 321D and outputs the detected inclination information to the busB2.

At step S254, the imaging data is transmitted as in the processing atstep S153 in FIG. 36. However, unlike the processing at step S153, theimaging data is transmitted using the packet in FIG. 44.

Thereafter, the processing of the imaging device 321D ends.

In this manner, each imaging device 321D detects its own position andattitude and notifies the signal processing device 332 of the same, sothat each imaging device 321D may be freely installed and moved toperform imaging processing without prior setting processing and thelike.

7. Fifth Embodiment

Next, a fifth embodiment of the present disclosure is described withreference to FIGS. 46 to 48.

Upper and lower stages of FIG. 46 illustrate light-receiving sensitivitycharacteristics to an incident angle of incident light of a certainpixel 121 a of an imaging element 121 as in the right part of FIG. 8.The upper stage of FIG. 46 illustrates the light-receiving sensitivitycharacteristic of the pixel 121 a before rotation, and the lower stageillustrates the light-receiving sensitivity characteristic of the pixel121 a after the pixel 121 a is rotated by an angle α around a referencepoint (0,0).

For example, the incident light incident on the pixel 121 a before therotation from a point light source P on a subject surface 31 at anincident angle (θx1, θy1) is incident on the pixel 121 a after therotation at an incident angle (θx2, θy2). Therefore, light-receivingsensitivity of the pixel 121 a to the incident light changes before andafter the rotation, and a detection signal level changes.

Therefore, it is necessary to adjust a coefficient also in a case whereorientation of the imaging device 321 (imaging element 121) changes.

Therefore, in the fifth embodiment, the orientation of each imagingdevice 321 is further detected by each imaging device 321, and a signalprocessing device 332 obtains the coefficient used for restoring arestored image on the basis of a detection result to restore therestored image.

<Configuration Example of Imaging Device 321E>

In the fifth embodiment, an imaging device 321E in FIG. 47 that is afifth embodiment of the imaging device 321 is used in the imaging system301 in FIG. 26. Note that, in FIG. 47, the same reference sign isassigned to a portion corresponding to that of the imaging device 321Din FIG. 41 and the description thereof is appropriately omitted.

The imaging device 321E is different from the imaging device 321D inthat a signal processing control unit 401E is provided in place of thesignal processing control unit 401D. The signal processing control unit401E is different from the signal processing control unit 401D in that adetection unit 611B is provided in place of the detection unit 611A.

The detection unit 611B is different from the detection unit 611A inthat a geomagnetism detection unit 623 is added.

The geomagnetism detection unit 623 detects orientation of the imagingdevice 321E such as east, west, south, and north. For example,orientation of a predetermined axis parallel to a light-receivingsurface of the imaging element 121 of the imaging device 321E isdetected as the orientation of the imaging device 321E. The geomagnetismdetection unit 623 outputs direction information indicating the detectedorientation to the bus B2.

FIG. 48 illustrates an example of a data configuration of a packet ofimaging data transmitted from each imaging device 321E.

The data configuration of the packet in FIG. 48 is different from thedata configuration of the packet in FIG. 44 in that a direction isadded.

As the direction, the orientation of the imaging device 321E detected bythe geomagnetism detection unit 623 of the imaging device 321E is set.

For example, in a case where the orientation of the imaging device 321Echanges by an angle α with respect to a case where this is installed inthe reference position, a change amount Δθx′ of an incident angle θx inan X-axis direction and a change amount Δθy′ of an incident angle θy ina Y-axis direction of a light beam on a light-receiving surface of animaging element 121 from each point light source of a subject surface tobe restored are expressed by following equations (11) and (12) by usingthe incident angle change amount Δθx of equation (9) and the incidentangle change amount Δθy of equation (10) described above.Δθx′=Δθx×cos α−Δθy×sin α  (11)Δθy′=Δθx×sin α+Δθy×cos α  (12)

Then, a restoration unit 501 of the signal processing device 332 obtainsa coefficient set group by using the incident angle change amount Δθx′in equation (11) and the incident angle change amount Δθy′ in equation(12) in place of the incident angle change amount Δθx in equation (9)and the incident angle change amount Δθy in equation (10) and restoresthe restored image by using the obtained coefficient set group.Therefore, the restored image is restored using a more appropriatecoefficient, and restoration accuracy and an image quality of therestored image are improved.

8. Sixth Embodiment

Next, a sixth embodiment of the present disclosure is described withreference to FIGS. 49 to 52.

<Configuration Example of Imaging Device 321F>

In the sixth embodiment of the present disclosure, an imaging device321F in FIG. 49 that is a sixth embodiment of the imaging device 321 isused in the imaging system 301 in FIG. 26. Note that, in FIG. 49, thesame reference sign is assigned to a portion corresponding to that ofthe imaging device 321E in FIG. 47 and the description thereof isappropriately omitted.

The imaging device 321F is different from the imaging device 321E inthat a signal processing control unit 401F is provided in place of thesignal processing control unit 401E. The signal processing control unit401F is different from the signal processing control unit 401E in that adetection unit 611C is provided in place of the detection unit 611B, anda drive unit 651 is added.

The detection unit 611C is different from the detection unit 611B inthat an altitude detection unit 624 is added.

The altitude detection unit 624 includes an altitude sensor, forexample. The altitude detection unit 624 detects an altitude of aposition in which the imaging device 321F is installed, and outputsaltitude information indicating the detected altitude to a bus B2.

The drive unit 651 includes, for example, a drive mechanism that moves(changes) the position and attitude of the imaging device 321F. Forexample, the drive unit 651 includes a wheel that moves the imagingdevice 321F on land, a propeller that moves the same in the air and thelike. Alternatively, for example, the drive unit 651 includes anactuator and the like that moves orientation and inclination of theimaging device 321F. With this drive unit 651, the imaging device 321Fmay change the position and attitude by itself.

<Processing of Imaging System 301>

Next, processing of the imaging system 301 is described with referenceto FIGS. 50 to 52.

<Processing of Signal Processing Unit 312>

First, processing of a signal processing unit 312 is described withreference to FIG. 50.

At step S301, the signal processing unit 312 transmits a drivingcommand.

Specifically, a control unit 502 of the signal processing device 332generates the driving command and codes the generated driving command.The driving command includes, for example, a movement instructionregarding movement of at least one of the position or attitude of theimaging device 321F to be driven, and an ID of the imaging device 321Fto be driven.

Note that, the movement instruction may be indicated by absoluteposition and attitude, or may be indicated by relative position andattitude. In the former case, for example, the position or attitude withwhich the imaging device 321F is moved is indicated by an absolutenumerical value (for example, latitude, longitude, direction, angle withrespect to a gravity direction and the like). In the latter case, forexample, a moving direction and a moving amount with respect to currentposition or attitude of the imaging device 321F are illustrated.

The control unit 502 supplies the coded driving command to areader/writer 331 via a communication unit 509.

The reader/writer 331 starts transmitting a carrier wave including anelectromagnetic wave, and transmits the driving command to each imagingdevice 321F by modulating the carrier wave with the coded drivingcommand.

Note that, one driving command may include the movement instructions fora plurality of imaging devices 321F, or the driving command may beindividually transmitted to each imaging device 321F to be driven. Inthe latter case, for example, when the positions or attitudes of theplurality of imaging devices 321F are changed, a plurality of drivingcommands is transmitted at step S301.

The imaging device 321F to be driven changes the position or attitude inaccordance with the instruction by the driving command at step S353 inFIG. 52 to be described later.

Note that, in a case where it is not necessary to change the positionand attitude of each imaging device 321F, the processing at step S301may be omitted.

At step S302, an imaging command is transmitted as in the processing atstep S101 in FIG. 34.

At step S303, imaging data is received from each imaging device 321F asin the processing at step S102 in FIG. 34.

However, a data configuration of a packet of the imaging data isdifferent from that in the processing at step S102 in FIG. 34.

FIG. 51 illustrates an example of the data configuration of the packetof the imaging data used in the sixth embodiment.

The data configuration of the packet in FIG. 51 is different from thedata configuration of the packet in FIG. 48 in that the altitude isadded.

As the altitude, the altitude of the imaging device 321F detected by thealtitude detection unit 624 of the imaging device 321F is set.

At step S304, as in the processing at step S203 in FIG. 43, acoefficient used for image restoration is obtained. However, arestoration unit 501 calculates an incident angle change amount Δθx andan incident angle change amount Δθy in further consideration of thealtitude of the imaging device 321F by using, for example, followingequations (13) and (14) in place of equations (9) and (10) describedabove.Δθx=atan((longitude after movement−reference longitude)/(subjectdistance−altitude))+Xθ  (13)Δθy=atan((latitude after movement−reference latitude)/(subjectdistance−altitude))+Yθ  (14)

Moreover, the restoration unit 501 calculates an incident angle changeamount Δθx′ and an incident angle change amount Δθy′ using equations(11) and (12) described above. Then, the restoration unit 501 reads outa coefficient set group corresponding to the set subject distance andthe incident angle change amounts Δθx′ and Δθy′ from a storage unit 506.

At step S305, as in the processing at step S104 in FIG. 34, imagerestoration is performed using a detection signal set and thecoefficient.

At step S306, as in the processing at step S105 in FIG. 34, varioustypes of processing are performed on the restored image.

Thereafter, the processing of the signal processing unit 312 ends.

<Processing of Imaging Device 321F>

Next, with reference to a flowchart in FIG. 52, processing executed byeach imaging device 321F corresponding to the processing of the signalprocessing unit 312 in FIG. 50 is described.

At step S351, the imaging device 321F receives the driving command.Specifically, an antenna 422 of the imaging device 321F receives theelectromagnetic wave of which transmission from the reader/writer 331 isstarted at step S301 in FIG. 50 and converts the receivedelectromagnetic wave into electric power to supply to each unit of theimaging device 321F. Therefore, each unit of the imaging device 321F isactivated.

Next, the transmission/reception unit 421 demodulates theelectromagnetic wave received from the reader/writer 331 via the antenna422, and supplies the coded data obtained as a result to the controlunit 411.

The control unit 411 obtains the driving command by decoding the codeddata.

At step S352, the control unit 411 determines whether or not there is aninstruction to change the position or attitude. For example, in a casewhere the ID of the imaging device to be driven included in the drivingcommand coincides with its own ID, the control unit 411 determines thatthere is the instruction to change the position or attitude, and theprocedure shifts to step S353.

At step S353, the imaging device 321F changes the position or attitude.Specifically, the control unit 411 controls the drive unit 651 to movethe position or attitude of the imaging device 321F in accordance withthe movement instruction included in the driving command.

Thereafter, the procedure shifts to step S354.

In contrast, at step S352, for example, in a case where the ID of theimaging device to be driven included in the driving command does notcoincide with its own ID, the control unit 411 determines that there isnot the instruction to change the position or attitude, the processingat step S353 is skipped, and the procedure shifts to step S354.

At step S354, the imaging command is received as in the processing atstep S151 in FIG. 36.

At step S355, a subject is imaged as in the processing at step S152 inFIG. 36.

At step S356, the detection unit 611C detects the position and attitude.Specifically, a position detection unit 621 detects the latitude andlongitude of the current position of the imaging device 321F, andoutputs the detected position information to the bus B2. An inclinationdetection unit 622 detects inclinations Xθ and Yθ of a light-receivingsurface of an imaging element 121 of the imaging device 321F, andoutputs the detected inclination information to the bus B2. Ageomagnetism detection unit 623 detects the orientation of the imagingdevice 321F and outputs the detected direction information to the busB2. The altitude detection unit 624 detects the altitude of the imagingdevice 321F and outputs the detected altitude information to the bus B2.

At step S357, the imaging data is transmitted as in the processing atstep S153 in FIG. 36. However, unlike the processing at step S153, theimaging data is transmitted using the packet in FIG. 51.

Thereafter, the processing of the imaging device 321F ends.

As described above, it is possible to image a desired subject whilemoving the position and attitude of each imaging device 321F by remoteoperation.

Furthermore, since the coefficient is obtained in further considerationof the altitude of the imaging device 321F, the restored image isrestored using a more appropriate coefficient, and restoration accuracyand an image quality of the restored image are improved.

Note that, although the example in which the driving command and theimaging command are continuously transmitted is described above, thedriving command and the imaging command may be transmitted at differenttimings. That is, the movement of the position or attitude of eachimaging device 321F and imaging processing may be performed at differenttimings.

Furthermore, for example, each imaging device 321F may autonomouslychange the position or attitude without being instructed by the signalprocessing unit 312. In this case, each imaging device 321F needs to beable to operate without receiving the electric power from thereader/writer 331.

Therefore, for example, the power supply unit 451 in FIG. 37 or thepower supply unit 461 in FIG. 38 may be provided in each imaging device321F. Alternatively, for example, an external DC power supply may bemounted on each imaging device 321F, or the electric power may beexternally supplied via a power supply line.

9. Seventh Embodiment

Next, with reference to FIG. 53, a seventh embodiment of the presentdisclosure is described.

<Configuration Example of Imaging Device 321G>

In the seventh embodiment, an imaging device 321G in FIG. 53 that is aseventh embodiment of the imaging device 321 is used as the imagingdevice 321 in the imaging system 301 in FIG. 26.

As described above, it is assumed that the imaging devices 321 aredisorderly distributed to be used. However, in a case where the imagingdevices 321 are distributed, a light-receiving surface 121A of animaging element 121 does not necessarily face up. For example, if thelight-receiving surface 121A of the imaging element 121 faces theground, an effective detection signal set cannot be obtained from theimaging device 321.

Therefore, for example, a countermeasure such as attaching a weight to aback surface of the imaging device 321 and the like is conceivable suchthat the light-receiving surface 121A of the imaging element 121 alwaysfaces up. However, even if the weight is attached to the imaging device321, the light-receiving surface 121A of the imaging element 121 doesnot always face up, and size and weight of the imaging device 321increase.

In contrast, in the imaging device 321G, in addition to thelight-receiving surface 121A of the imaging element 121 on a frontsurface, a light-receiving surface 701A of an imaging element 701 (notillustrated) having a configuration similar to that of the imagingelement 121 is provided on a back surface (surface opposite to the frontsurface). Therefore, the imaging device 321F may image a subject andobtain the effective detection signal set regardless of the surface outof the front surface and the back surface that faces the front.

Note that, the imaging device 321G may determine which of the imagingelement 121 and the imaging element 701 is effective, and may transmitimaging data including only the detection signal set of the imagingelement determined to be effective. For example, the imaging device 321Gmay determine that the imaging element having a larger detection signallevel average value out of the detection signal sets of the two imagingelements is effective, and transmit the imaging data including only thedetection signal set of the imaging element determined to be effective.

Alternatively, the imaging device 321G may always transmit the imagingdata including the detection signal sets of both the imaging elements,and the signal processing device 332 may determine which detectionsignal set is effective.

Furthermore, the imaging element 121 and the imaging element 701 may beformed using one imaging element. In this case, for example, the imagingelement has a configuration bent at 180°, and a pixel array unit on onesurface and the pixel array unit on the other surface may operateindependently.

Moreover, the imaging device 321F may be formed by using a polyhedronhaving three or more surfaces with different directions such as aregular tetrahedron or a regular hexahedron, and the light-receivingsurface of the imaging element may be provided on each surface.

10. Variation

Hereinafter, a variation of the embodiments of the present disclosuredescribed above is described.

<Variation regarding Imaging Device and Imaging Element>

Characteristics of the imaging devices 321A to 321F described above maybe combined in a possible range.

For example, the altitude detection unit 624 of the imaging device 321Fin FIG. 49 may be added to the imaging device 321E in FIG. 47.Alternatively, for example, the imaging device 321D in FIG. 41, theimaging device 321E in FIG. 47, or the imaging device 321F in FIG. 49may be provided with the power supply unit 451 of the imaging device321B in FIG. 37 or the power supply unit 461 of the imaging device 321Cin FIG. 38.

Furthermore, for example, in a case where the imaging device 321 isdriven by electric power supplied from a reader/writer 331 or electricpower generated by solar power generation, a case where sufficientelectric power cannot be obtained is assumed. For this, in a case wherethe sufficient electric power cannot be obtained, some of pixels 121 amay be made ineffective.

For example, upper and lower stages of FIG. 54 illustrate examples of apattern of a pixel array unit of an imaging element 121. In thisexample, a pixel block of one row×four columns of an R pixel providedwith a red color filter, a G pixel provided with a green color filter, aB pixel provided with a blue color filter, and a W pixel provided with awhite color filter is made one unit, and the pixel blocks are arrangedin three rows×two columns. Furthermore, a black portion of each pixel121 a indicates a light-shielding film 121 b. Note that, in thisexample, a light-shielding range of the light-shielding film 121 b ofall the pixels 121 a is set to the same range, but this is notnecessarily set to the same range.

For example, in a case where the electric power supply is sufficient,all the pixels 121 a are made effective as illustrated in the upperstage. That is, detection signals of all the pixels 121 a are read outto be externally transmitted.

On the other hand, in a case where the electric power supply isinsufficient, some pixels 121 a are made ineffective as illustrated inthe upper stage. In this example, the pixels 121 a in five pixel blocksenclosed by a bold frame are made ineffective.

Here, to make the pixel 121 a ineffective is, for example, to stopreading out the detection signal of the pixel 121 a. For example, bystopping AD conversion of the detection signal of the ineffective pixel121 a, readout of the detection signal is stopped.

Note that, the detection signal of the ineffective pixel 121 a may beexternally transmitted or not. In a case where this is externallytransmitted, for example, a signal having a predetermined dummy value istransmitted as the detection signal of the ineffective pixel 121 a.

Therefore, even in a case where the electric power supply isinsufficient, a part of the detection signals may be supplied to asignal processing device 332, and restoration of the image may becontinued.

Note that, the number of pixels 121 a made ineffective may be changedaccording to an amount of power shortage.

Furthermore, for example, a shape other than the above-described lateralband-type, longitudinal band-type, L-shaped type, and type provided witha rectangular opening may be adopted as the shape of the light-shieldingfilm 121 b of each pixel 121 a.

Moreover, for example, in the imaging element 121 described above withreference to FIG. 5, the example in which four photodiodes 121 f of tworows×two columns are provided in one pixel 121 a is illustrated, but thenumber and arrangement of the photodiodes 121 f are not limited to thisexample.

For example, as illustrated in FIG. 55, in one pixel 121 a, ninephotodiodes 121 f-111 to 121 f-119 arranged in three rows×three columnsmay be provided, for example, for one on-chip lens 121 c. That is, onepixel output unit may include the nine photodiodes 121 f.

Then, for example, by not reading out the signals of the five pixels ofthe photodiodes 121 f-111, 121 f-114, and 121 f-117 to 121 f-119, anincident angle characteristic similar to that of the pixel 121 aincluding the L-shaped light-shielding film 121 b in which thelight-shielding film 121 b is set in a range of the photodiodes 121f-111, 121 f-114, and 121 f-117 to 121 f-119 may be substantiallyobtained.

In this manner, it is possible to obtain the incident anglecharacteristic similar to that in a case where the light-shielding film121 b is provided without providing the light-shielding film 121 b.Furthermore, by switching a pattern of the photodiodes 121 f from whichno signal is read out, incident angle directivity may be changed as in acase where the position and range light-shielded by the light-shieldingfilm 121 b are changed.

Furthermore, in the above description, the example in which one pixel121 a forms one pixel output unit is illustrated; however, it is alsopossible that a plurality of pixels 121 a forms one pixel output unit.

For example, as illustrated in FIG. 56, the pixels 121 a-111 to 121a-119 arranged in three rows×three columns may form one pixel outputunit 801 b. Note that, each of the pixels 121 a-111 to 121 a-119includes, for example, one photodiode and does not include the on-chiplens.

For example, by adding a pixel signal from each pixel 121 a, thedetection signal of one pixel of a detection image may be generated, andby stopping or not adding an output of pixel signals from some pixels121 a, the incident angle directivity of a pixel output unit 801 b maybe realized. For example, by adding the pixel signals of the pixels 121a-112, 121 a-113, 121 a-115, and 121 a-116 to generate the detectionsignal, the incident angle directivity similar to that in a case ofproviding the L-shaped light-shielding film 121 b in a range of thepixels 121 a-111, 121 a-114, and 121 a-117 to 121 a-119 may be obtained.

Furthermore, by switching the pattern of the pixel 121 a the pixelsignal of which is added to the detection signal, the incident angledirectivity may be set to a different value as in a case where theposition and range light-shielding by the light-shielding film 121 b arechanged.

Furthermore, in this case, for example, it is possible to change therange of the pixel output unit by changing a combination of the pixels121 a. For example, the pixels 121 a of two rows×two columns includingthe pixels 121 a-111, 121 a-112, 121 a-114, and 121 a-115 may form apixel output unit 801 s.

Moreover, for example, by recording the pixel signals of all the pixels121 a and later setting the combination of the pixels 121 a, it ispossible to set the range of the pixel output unit later. Moreover, byselecting the pixel 121 a the pixel signal of which is added to thedetection signal out of the pixels 121 a in the set pixel output unit,the incident angle directivity of the pixel output unit may be setlater.

Furthermore, although the example of providing the different incidentangle directivities to the respective pixels by using thelight-shielding film 121 b as the modulation element and changing thecombination of the photodiodes contributing to the output is illustratedin FIG. 4, in the present disclosure, it is also possible to provide theincident angle directivity to each pixel by using an optical filter 902covering a light-receiving surface of an imaging element 901 as themodulation element as illustrated in FIG. 57, for example.

Specifically, the optical filter 902 is arranged so as to cover anentire surface of a light-receiving surface 901A at a predeterminedinterval from the light-receiving surface 901A of the imaging element901. Light from a subject surface 31 is modulated by the optical filter902 to be incident on the light-receiving surface 901A of the imagingelement 901.

For example, as the optical filter 902, an optical filter 902BW having ablack and white lattice pattern illustrated in FIG. 58 may be used. Inthe optical filter 902BW, a white pattern portion that transmits lightand a black pattern portion that blocks light are randomly arranged. Asize of each pattern is set independently of a pixel size of the imagingelement 901.

FIG. 59 illustrates a light-receiving sensitivity characteristic of theimaging element 901 to light from a point light source PA and light froma point light source PB on the subject surface 31 in a case where theoptical filter 902BW is used. The light from the point light source PAand the light from the point light source PB are modulated by theoptical filter 902BW to be incident on the light-receiving surface 901Aof the imaging element 901.

For example, the light-receiving sensitivity characteristic of theimaging element 901 to the light from the point light source PA is as awaveform Sa. That is, since a shadow is generated by the black patternportion of the optical filter 902BW, a shaded pattern is generated in animage on the light-receiving surface 901A for the light from the pointlight source PA. Similarly, the light-receiving sensitivitycharacteristic of the imaging element 901 to the light from the pointlight source PB is as a waveform Sb. That is, since a shadow isgenerated by the black pattern portion of the optical filter 902BW, ashaded pattern is generated in an image on the light-receiving surface901A for the light from the point light source PB.

Note that, the light from the point light source PA and the light fromthe point light source PB have different incident angles with respect toeach white pattern portion of the optical filter 902BW, so that there isa shift in appearance of the shaded pattern on the light-receivingsurface. Therefore, each pixel of the imaging element 901 has theincident angle directivity to each point light source on the subjectsurface 31.

This method is disclosed in detail, for example, in Non-Patent Document1 described above.

Note that, an optical filter 902HW in FIG. 60 may be used in place ofthe black pattern portion of the optical filter 902BW. The opticalfilter 902HW includes linear polarizing elements 911A and 911B havingthe same polarizing direction, and a half-wavelength plate 912, and thehalf-wavelength plate 912 is interposed between the linear polarizingelements 911A and 911B. The half-wavelength plate 912 includes apolarizing portion indicated by oblique lines in place of the blackpattern portion of the optical filter 902BW, and the white patternportion and the polarizing portion are randomly arranged.

The linear polarizing element 911A transmits only light in apredetermined polarizing direction out of substantially non-polarizedlight emitted from the point light source PA. Hereinafter, it is assumedthat the linear polarizing element 911A transmits only light thepolarizing direction of which is parallel to the drawing. Out ofpolarized light transmitted through the linear polarizing element 911A,the polarized light transmitted through the polarizing portion of thehalf-wavelength plate 912 is such that a polarizing surface is rotatedand the polarizing direction changes in a direction perpendicular to thedrawing. On the other hand, out of the polarized light transmittedthrough the linear polarizing element 911A, the polarized lighttransmitted through the white pattern portion of the half-wavelengthplate 912 is such that the polarizing direction remains unchanged fromthe direction parallel to the drawing. Then, the linear polarizingelement 911B transmits the polarized light transmitted through the whitepattern portion and hardly transmits the polarized light transmittedthrough the polarizing portion. Accordingly, an amount of polarizedlight transmitted through the polarizing portion is reduced as comparedto the polarized light transmitted through the white pattern portion.Therefore, a shaded pattern substantially similar to that in a case ofusing the optical filter BW is generated on the light-receiving surface901A of the imaging element 901.

Furthermore, as illustrated in FIG. 61A, an optical interference maskmay be used as an optical filter 902LF. The light emitted from the pointlight source PA and the light emitted from the point light source PB ofthe subject surface 31 are emitted to the light-receiving surface 901Aof the imaging element 901 via the optical filter 902LF. As illustratedin an enlarged view in a lower portion in A of FIG. 61A, for example, alight incident surface of the optical filter 902LF includesirregularities of about a wavelength. Furthermore, in the optical filter902LF, transmission of light having a specific wavelength emitted in thevertical direction is the maximum. When a change in incident angle(inclination with respect to the vertical direction) of the light beamshaving the specific wavelength emitted from the point light sources PAand PB of the subject surface 31 on the optical filter 902LF increases,an optical path length changes. Here, when the optical path length is anodd multiple of the half wavelength, the light beams weaken each other,and when this is an even multiple of the half wavelength, the lightbeams strengthen each other. That is, intensities of the transmittedlight beams having the specific wavelength emitted from the point lightsources PA and PB and transmitted through the optical filter 902LF aremodulated according to the incident angle with respect to the opticalfilter 902LF to be incident on the light-receiving surface 901A of theimaging element 901 as illustrated in FIG. 61B. Therefore, the detectionsignal output from each pixel output unit of the imaging element 901 isa signal obtained by combining the modulated light intensities of thepoint light sources for each pixel output unit.

This method is disclosed in detail, for example, in Patent Document 1described above.

Note that, in the methods of Patent Document 1 and Non-Patent Document1, it is not possible to independently set the incident angledirectivity in a pixel 121 a unit without affecting adjacent pixels asthe imaging element 121 using the pixel 121 a in FIG. 4 or the pixel 121a in FIG. 5 described above. Therefore, for example, when the pattern ofthe optical filter 902BW or the pattern of a diffraction grating of theoptical filter 902LF is different, the incident angle directivities ofat least a plurality of adjacent pixels of the imaging element 901 aredifferent from each other. Furthermore, the pixels 121 a located atclose positions have incident angle directivities close to each other.

Furthermore, the present disclosure is also applicable to the imagingdevice and imaging element that image light of a wavelength other thanvisible light such as infrared light. In this case, the restored imageis not the image in which the user may visually recognize the subjectbut the image in which the user cannot visually recognize the subject.Note that, since it is difficult for a normal imaging lens to transmitfar-infrared light, the present technology is effective in a case ofimaging the far-infrared light, for example. Therefore, the restoredimage may be a far-infrared light image, and may be a visible lightimage or a non-visible light image in addition to the far-infrared lightimage.

<Variation regarding Signal Processing Unit>

For example, an association unit 504 of the signal processing device 332may associate a detection signal set obtained from each imaging device321 with metadata corresponding to each detection signal set. In thiscase, for example, the association unit 412 in FIGS. 28, 37, 38, 41, 47,and 49 may be omitted.

Furthermore, the metadata may include a coefficient set group at thetime of restoration or not, for example. In the latter case, forexample, data indicating a subject distance and an angle of view at thetime of restoration and a position and attitude at the time of imagingof each imaging device 321 (for example, latitude, longitude,inclination Xθ, inclination Yθ, orientation, altitude and the like) isincluded in the metadata.

Note that, a method of associating the detection signal set with themetadata is not especially limited as long as a correspondencerelationship between the detection signal set and the metadata may bespecified. For example, by assigning the metadata to data including thedetection signal set, assigning the same ID to the detection signal setand the metadata, or recording the detection signal set and the metadataon the same recording medium 508, the detection signal set and themetadata are associated with each other. Furthermore, the metadata maybe associated with each detection signal set individually, or themetadata may be associated with data in which the detection signal setsare combined into one.

Moreover, it is not always necessary to use the detection signal sets ofall the imaging devices 321 for restoring the restored image; forexample, a restoration unit 501 may restore the restored image using thedetection signal sets of some imaging devices 321.

For example, the imaging element 121 suitable for imaging a subjectsurface with a different subject distance or angle of view may beprovided for each imaging device 321. Then, for example, in a case wherethe subject distance and the angle of view may be specified, therestoration unit 501 may restore the restored image by using only thedetection signal set of the imaging device 321 suitable for the imagingof the specified subject distance and angle of view without using thedetection signal sets of all of the imaging devices 321. In this case,it is not always necessary to make the appropriate subject distance andangle of view different in all the imaging devices 321, and they may bethe same in some imaging devices 321.

Furthermore, for example, the restoration unit 501 may select thedetection signal set used for restoring the restored image on the basisof the restored image that is simply restored (hereinafter referred toas a simple restored image).

Specifically, first, the restoration unit 501 simply restores therestored image using the detection signal sets of all the imagingdevices 321. For example, the restoration unit 501 thins out thedetection signal of a predetermined pixel 121 a among the detectionsignal of each pixel 121 a included in each detection signal set, andsimply performs the restoration processing with a small number ofpixels. Alternatively, the restoration unit 501 performs the restorationprocessing simply, for example, by limiting an arithmetic amount (forexample, the number of times of iterative operations and the like).

Next, the restoration unit 501 sets a main subject on the basis of, forexample, the simply restored image. A method of setting the main subjectis arbitrary. For example, the restoration unit 501 sets the largestsubject in the simply restored image or a predetermined type of subject(for example, a person and the like) in the simply restored image as themain subject. Alternatively, for example, the user may visuallyrecognize the simply restored image and set a desired subject in thesimply restored image as the main subject.

Next, the restoration unit 501 selects a detection signal set used forrestoration of the restored image.

Specifically, an imaging range in the real world of each pixel 121 a ofeach imaging device 321 may be estimated on the basis of, for example,the position and attitude of each imaging device 321 and thelight-shielding range of each pixel 121 a. Furthermore, the position ofthe main subject in the real world may be estimated on the basis of theposition and the like of the main subject in the simply restored image,for example. Therefore, the restoration unit 501 extracts the detectionsignal set including the main subject (the main subject is imaged) outof the detection signal sets of all the imaging devices 321 on the basisof the imaging range in the rear world of each pixel 121 a and theposition in the real world of the main subject and selects the same asthe detection signal set to be used for restoring the restored image.

Alternatively, for example, the restoration unit 501 selects the imagingelement 121 that serves as a master (hereinafter referred to as a mastersensor). For example, the restoration unit 501 estimates the size andposition of the main subject within the imaging range of the imagingelement 121 of each imaging device 321 on the basis of the imaging rangeof each pixel 121 a in the real world and the position of the mainsubject in the real world. Then, the restoration unit 501 selects theimaging element 121 with the most appropriate size and position of themain subject as the master sensor.

Note that, the appropriate size and position of the main subject varydepending on the type of the main subject, the scene to be captured andthe like.

Next, the restoration unit 501 extracts, for example, the imagingelement 121 in which a ratio of a region where the imaging rangeoverlaps the imaging range of the master sensor is equal to or largerthan a predetermined threshold from the imaging elements 121 except themaster sensor. The restoration unit 501 selects the detection signal setof the master sensor and the detection signal set of the extractedimaging element 121 as the detection signal set used for restoration ofthe restored image.

Then, the restoration unit 501 performs the restoration processing ofthe restored image using the selected detection signal set. For example,the restoration unit 501 restores the restored image by creating andsolving the simultaneous equations described above using the coefficientset corresponding to each pixel 121 a of each imaging device 321 fromwhich the selected detection signal set is obtained.

Therefore, it is possible to reduce the load of the restorationprocessing while maintaining an excellent image quality (for example,resolution and the like) of the main subject in the restored image.

Note that, the pixel 121 a used for restoration of the restored imagemay be further selected from the pixels 121 a of the selected imagingdevice 321. For example, the restoration unit 501 may select thedetection signal of the pixel 121 a whose imaging range overlaps themain subject from the selected detection signal set, and restore therestored image using the selected detection signal.

Furthermore, in the above description, the example in which a coordinatesystem (hereinafter referred to as a reference coordinate system) usedfor the calculation of equations (9) to (14) is set on the basis of apredetermined reference position (reference latitude, referencelongitude) is described; however, the reference position may bevariable.

For example, the reference position may be set on the basis of theposition of the main subject. For example, the position in the realworld of a predetermined position (for example, the center of gravity)of the main subject may be set as the reference position.

Alternatively, for example, the reference position may be set on thebasis of the position of the master sensor. For example, the pixel at apredetermined position (for example, center, upper left corner and thelike) of the master sensor may be set as the reference position.

In this case, for example, the reference coordinate system may be set onthe basis of the master sensor. For example, on the basis of thecoordinate system of a pixel region of the master sensor, X and Y-axesof the reference coordinate system may be set, and an axis perpendicularto a light-receiving surface of the master sensor may be set as a Z-axisof the reference coordinate system.

Moreover, for example, the subject distance and the angle of view of therestored image may be set on the basis of the main subject. For example,the subject distance of the restored image may be set so that the imagequality of the main subject is the best. Furthermore, for example, theangle of view of the restored image may be set so that the main subjecthas an appropriate size in the restored image.

Note that, for example, in a case where the imaging system 301 isdesigned on the assumption of infinity, the subject distance is alwaysset to infinity.

<Variation regarding System Configuration>

The configuration of the imaging system 301 in FIG. 26 may be changed.

For example, the reader/writer 331 and the signal processing device 332may be integrated, or a part of the functions of the signal processingdevice 332 may be provided in the reader/writer 331.

Furthermore, the function of the reader/writer 331 and the function ofthe signal processing device 332 may be provided in some imagingdevices.

FIG. 62 illustrates a configuration example of an imaging system 1001that is a first variation of the imaging system. Note that, in thedrawing, a portion corresponding to that of the imaging system 301 inFIG. 26 is assigned with the same reference sign, and the descriptionthereof is omitted as appropriate.

The imaging system 1001 is different from the imaging system 301 in thatan imaging unit 1011 is provided in place of the imaging unit 311 andthe reader/writer 331 is deleted. The imaging unit 1011 is differentfrom the imaging unit 311 in that an imaging device 1021 is added.

The imaging device 1021 has the function of the reader/writer 331. Thatis, the imaging device 1021 performs short-range wireless communicationwith each imaging device 321 and supplies electric power to each imagingdevice 321. Then, the imaging device 1021 transmits an imaging commandto each imaging device 321, controls imaging of a subject by eachimaging device 321, and also images the subject by itself. Then, theimaging device 1021 receives imaging data from each imaging device 321,and transmits the imaging data received from each imaging device 321 andimaging data obtained by imaging by itself to the signal processingdevice 332. That is, the imaging device 1021 plays a role of relayingthe imaging data from each imaging device 321 to the signal processingdevice 332.

Note that, the imaging device 1021 includes, for example, a power supplyunit similar to the power supply unit 451 of the imaging device 321B inFIG. 37 or the power supply unit 461 of the imaging device 321C in FIG.38, and may operate without electric power supplied from thereader/writer 331.

Furthermore, in the imaging system 1001, the number of imaging devices321 may be one.

FIG. 63 illustrates a configuration example of an imaging system 1101that is a second variation of the imaging system. Note that, in thedrawing, a portion corresponding to that of the imaging system 301 inFIG. 26 is assigned with the same reference sign, and the descriptionthereof is omitted as appropriate.

The imaging system 1101 is different from the imaging system 301 in thatan imaging unit 1111 is provided in place of the imaging unit 311 andthe reader/writer 331 and the signal processing device 332 are deleted.The imaging unit 1111 is different from the imaging unit 311 in that animaging device 1121 is added.

The imaging device 1121 has a function similar to that of the imagingdevice 1021 in FIG. 62 and a function similar to that of the signalprocessing device 332 in FIG. 26. Then, the imaging device 1121 performsrestoration processing and the like of a restored image on the basis ofthe imaging data received from each imaging device 321 and the imagingdata obtained by imaging by itself.

Note that, in the imaging system 1101, the number of imaging devices 321may be one.

<Other Variations>

Furthermore, in the above description, the example in which each imagingdevice 321 detects its own position and attitude and notifies the signalprocessing unit 312 of the detection result; however, for example, theposition and attitude of each imaging device 321 may be externallydetected. For example, a region where each imaging device 321 isinstalled may be externally imaged, and the position and attitude ofeach imaging device 321 may be detected on the basis of the capturedimage. Furthermore, for example, the reader/writer 331 or the signalprocessing device 332 may have such a detecting function.

Moreover, in the above-described example, the example of detecting theposition (absolute position), inclination, orientation, and altitude asthe position and attitude of the imaging device 321 is described, butthe combination may be freely changed. Furthermore, another dataindicating the position or attitude of the imaging device 321 may bedetected. Moreover, for example, only one of the position or attitude ofthe imaging device 321 may be detected.

Furthermore, for example, by applying machine learning such as deeplearning, it is also possible to perform image recognition and the likeusing the detection image before restoration and the detection signalset without using the restored image after the restoration. In this casealso, accuracy of image recognition using the detection image before therestoration is improved by using the present technology. In other words,the image quality of the detection image before the restoration isimproved.

11. Other

The above-described series of processes may be executed by hardware ormay be executed by software. In a case where a series of processes isexecuted by the software, a program which forms the software isinstalled on a computer. Here, the computer includes a computer (forexample, the control unit 123 and the like) incorporated in dedicatedhardware, for example.

The program executed by the computer may be recorded in a recordingmedium (for example, the recording medium 130 and the like) as a packagemedium and the like to be provided, for example. Furthermore, theprogram may be provided by means of a wired or wireless transmissionmedium such as a local area network, the Internet, and digitalbroadcasting.

Note that, the program executed by the computer may be the program ofwhich processes are performed in chronological order in the orderdescribed in this specification or may be the program of which processesare performed in parallel or at required timing such as when a call isissued.

Furthermore, in this specification, a system is intended to meanassembly of a plurality of components (devices, modules (parts) and thelike) and it does not matter whether or not all the components are inthe same casing. Therefore, a plurality of devices stored in differentcasings and connected through a network and one device obtained bystoring a plurality of modules in one casing are the systems.

Moreover, the embodiments of the present technology are not limited tothe above-described embodiments and various modifications may be madewithout departing from the gist of the present technology.

For example, the present technology may be configured as cloud computingin which a function is shared by a plurality of devices through thenetwork to process together.

Furthermore, each step described in the above-described flowchart may beexecuted by one device or executed by a plurality of devices in a sharedmanner.

Moreover, in a case where a plurality of processes is included in onestep, a plurality of processes included in one step may be executed byone device or by a plurality of devices in a shared manner.

Note that, the present disclosure may also have the followingconfiguration.

(1)

An imaging device including:

an imaging element that includes one or more pixel output units thatreceive incident light from a subject incident without an interventionof an imaging lens or a pinhole and output one detection signalindicating an output pixel value modulated by an incident angle of theincident light, and outputs a detection signal set including one or moredetection signals; and

a communication unit that transmits imaging data including the detectionsignal set and position attitude data indicating at least one of aposition or an attitude to a communication device by wirelesscommunication.

(2)

The imaging device according to (1) described above,

in which at least one of the pixel output units has a configurationcapable of independently setting incident angle directivity indicatingdirectivity to the incident angle of the incident light.

(3)

The imaging device according to (2) described above,

in which the imaging element includes a plurality of the pixel outputunits having different detection wavelengths.

(4)

The imaging device according to (3) described above, in which incidentangle directivities of the respective pixel output units are same.

(5)

The imaging device according to (2) described above, in which theimaging element includes a plurality of the pixel output units having asame detection wavelength, and incident angle directivities of therespective pixel output units are different from each other.

(6)

The imaging device according to any one of (1) to (5) described above,further including:

a detection unit that detects at least one of the position or theattitude of the imaging device.

(7)

The imaging device according to (6) described above,

in which the position attitude data includes at least one of an absoluteposition, inclination, orientation, or altitude of the imaging device.

(8)

The imaging device according to (6) or (7) described above, furtherincluding:

a drive unit that changes at least one of the position or the attitudeof the imaging device by an instruction transmitted from thecommunication device.

(9)

The imaging device according to any one of (1) to (8) described above,

in which the imaging data includes identification information foridentifying the imaging device.

(10)

The imaging device according to any one of (1) to (9) described above,further including:

a plurality of light-receiving surfaces with different directions.

(11)

The imaging device according to (10) described above, further including:

a plurality of imaging elements that includes the light-receivingsurfaces, respectively.

(12)

The imaging device according to (11) described above, in which thecommunication unit transmits imaging data including the detection signalof an effective imaging element out of the plurality of imagingelements.

(13)

The imaging device according to any one of (10) to (12) described above,further including:

a first light-receiving surface provided on a first surface of theimaging device; and

a second light-receiving surface provided on a second surface oppositeto the first surface of the imaging device.

(14)

The imaging device according to any one of (1) to (13) described above,

in which the communication unit receives electric power supplied by anelectromagnetic wave from the communication device.

(15)

The imaging device according to (14) described above, in which theimaging element includes two or more of the pixel output units, andmakes a part of the pixel output units ineffective according to theelectric power supplied from the communication device.

(16)

The imaging device according to any one of (1) to (13) described above,further including:

a power supply unit that supplies electric power for driving the imagingelement and the communication unit.

(17)

The imaging device according to (16) described above, in which the powersupply unit has a power generating function.

(18)

The imaging device according to (17) described above, in which the powersupply unit generates power by sunlight, and

a photoelectric conversion unit of the power supply unit and aphotoelectric conversion unit of the imaging element are formed on asame semiconductor substrate.

(19)

A signal processing device including:

a restoration unit that restores a restored image by using a pluralityof detection signal sets included in a plurality of imaging data from aplurality of imaging devices each including: an imaging element thatincludes one or more pixel output units that receive incident light froma subject incident without an intervention of an imaging lens or apinhole and output one detection signal indicating an output pixel valuemodulated by an incident angle of the incident light, and outputs adetection signal set including one or more of the detection signals.

(20)

The signal processing device according to (19) described above,

in which each of the imaging data includes position attitude dataindicating at least one of a position or an attitude of each of theimaging devices, and

the restoration unit restores the restored image by further using theposition attitude data of each of the imaging devices.

(21)

The signal processing device according to (19) or (20) described above,

in which the restoration unit restores the restored image by using thedetection signal of each of the pixel output units and a plurality ofcoefficients indicating directivity of each of the pixel output units toan incident angle of incident light from the subject.

(22)

The signal processing device according to any one of (19) to (21),further including:

a communication unit that receives the plurality of imaging data fromthe plurality of imaging devices by wireless communication and supplieselectric power to one or more of the plurality of imaging devices by anelectromagnetic wave.

Note that, the effects described in this specification are illustrativeonly and are not limitative; there may also be another effect.

REFERENCE SIGNS LIST

-   101 Imaging device-   111 Signal processing control unit-   121 Imaging element-   121 a, 121 a′ Pixel-   121A Light-receiving surface-   121 b Light-shielding film-   121 c On-chip lens-   121 e, 121 f Photodiode-   122 Restoration unit-   123 Control unit-   125 Detection unit-   126 Association unit-   301 Imaging system-   311 Imaging unit-   312 Signal processing unit-   321, 321A to 321G Imaging device-   331 Reader/writer-   332 Signal processing device-   401A to 401F Signal processing control unit-   411 Control unit-   412 Association unit-   414 Communication unit-   421 Transmission/reception unit-   422 Antenna-   451, 461 Power supply unit-   471 Photoelectric conversion unit-   472 Power storage unit-   501 Restoration unit-   502 Control unit-   504 Association unit-   611A to 611C Detection unit-   621 Position detection unit-   622 Inclination detection unit-   623 Geomagnetism detection unit-   624 Altitude detection unit-   651 Drive unit-   701 Imaging element-   701A Light-receiving surface-   801 b, 801 s Pixel output unit-   901 Imaging element-   901A Light-receiving surface-   902, 902BW, 902F Optical filter-   1001 Imaging system-   1011 Imaging unit-   1021 Imaging device-   1101 Imaging system-   1111 Imaging unit-   1121 Imaging device

The invention claimed is:
 1. A signal processing device, comprising:circuitry configured to restore a restored image, which is a visibleimage upon display, based on a detection signal set, which is notvisible image that does not represent a captured scene, including aplurality of detection signals and an incident angle coefficient setdetermined based on an incident angle change amount and a referencecoefficient set of a reference position, wherein the detection signalset is output from an imaging element that includes a plurality of pixeloutput units, each of the plurality of pixel output units receivingincident light from a subject incident without an intervention of bothof an imaging lens and a pinhole and outputting one detection signalindicating an output pixel value modulated based on an incident angle ofthe incident light and thereby the detection signal set including theplurality of detection signals is output from the imaging element. 2.The signal processing device according to claim 1, wherein each imagingdata includes position attitude data indicating at least one of aposition or an attitude of each of imaging devices, and the circuitry isfurther configured to restore the restored image by using the positionattitude data of each of the imaging devices.
 3. The signal processingdevice according to claim 1, wherein the circuitry is further configuredto: execute wireless communication with imaging devices, wherein each ofthe imaging devices includes the imaging element; and supply electricpower to each at least one of the plurality of imaging devices by anelectromagnetic wave.
 4. The signal processing device according to claim1, wherein the circuitry is further configured to calculate the incidentangle coefficient set according to the incident angle change amount andthe reference coefficient set of the reference position.
 5. The signalprocessing device according to claim 4, wherein the circuitry is furtherconfigured to set the reference position based on a position of a mainsubject.
 6. The signal processing device according to claim 4, whereinthe circuitry is further configured to set the reference position basedon a position of a master sensor selected from a plurality of imagingelements.
 7. The signal processing device according to claim 1, whereinthe circuitry is further configured to restore the restored image basedon the detection signal set output from the imaging element of each ofimaging devices.
 8. The signal processing device according to claim 1,wherein the circuitry is further configured to set a main subject basedon a simply restored image.
 9. The signal processing device according toclaim 1, further comprising the imaging element.
 10. A method executedby a signal processing device, comprising: restoring a restored image,which is a visible image upon display, based on a detection signal set,which is not visible image that does not represent a captured scene,including a plurality of detection signals and an incident anglecoefficient set determined based on an incident angle change amount anda reference coefficient set of a reference position, wherein thedetection signal set is output from an imaging element that includes aplurality of pixel output units, each of the plurality of pixel outputunits receiving incident light from a subject incident without anintervention of both of an imaging lens and a pinhole and outputting onedetection signal indicating an output pixel value modulated based on anincident angle of the incident light and thereby the detection signalset including the plurality of detection signals is output from theimaging element.
 11. A non-transitory computer-readable medium havingstored thereon, computer-executable instructions which, when executed bya computer, cause the computer to execute operations, the operationscomprising: restoring a restored image, which is a visible image upondisplay, based on a detection signal set, which is not visible imagethat does not represent a captured scene, including a plurality ofdetection signals and an incident angle coefficient set determined basedon an incident angle change amount and a reference coefficient set of areference position, wherein the detection signal set is output from animaging element that includes a plurality of pixel output units, each ofthe plurality of pixel output units receiving incident light from asubject incident without an intervention of both of an imaging lens anda pinhole and outputting one detection signal indicating an output pixelvalue modulated based on an incident angle of the incident light andthereby the detection signal set including the plurality of detectionsignals is output from the imaging element.